2024-02-20T07:39:00.294 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-20T07:39:00.295 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_pdonnell@teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543 -- /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543/orig.config.yaml 2024-02-20T07:39:00.349 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T07:39:00.449 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543 branch: wip-batrick-testing-20240220.022152 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7568543' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240220.022152 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240220.022152 sha1: 3d72bcfd7affbb4a923c191d212814b4934cfeaa owner: scheduled_pdonnell@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9910 sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: wip-batrick-testing-20240220.022152 suite_path: /home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 3d72bcfd7affbb4a923c191d212814b4934cfeaa targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhw/JKRnkaJADFLDR3Xa7VLMAIBk8pQK5DHw2bEuYRAlZcpIRX+cmrW8BQEotiKhl8n9TUmjY4AFghGAYsFqGY= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6cYsuhI8MozMWzH0Sjph1vRRLMXMVXpIMfcT2j0scvrS/+Bid6b3tOdWStzxUFp6jvMOSl44DdhGZSejjjL5s= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - full_sequential_finally: - sleep: duration: 3600 - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-20_07:23:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-20T07:39:00.449 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa; will attempt to use it 2024-02-20T07:39:00.450 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks 2024-02-20T07:39:00.450 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-20T07:39:00.451 INFO:teuthology.task.internal:Checking packages... 2024-02-20T07:39:00.478 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '91ff50e903dcc99ee3ecf9523050878eb452da5c' 2024-02-20T07:39:00.479 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:39:00.479 INFO:teuthology.packaging:ref: None 2024-02-20T07:39:00.479 INFO:teuthology.packaging:tag: None 2024-02-20T07:39:00.479 INFO:teuthology.packaging:branch: wip-batrick-testing-20240220.022152 2024-02-20T07:39:00.479 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:39:00.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240220.022152 2024-02-20T07:39:00.619 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1587.g91ff50e9 2024-02-20T07:39:00.620 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-20T07:39:00.627 INFO:teuthology.task.internal:no buildpackages task found 2024-02-20T07:39:00.627 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-20T07:39:00.633 INFO:teuthology.task.internal:Saving configuration 2024-02-20T07:39:00.655 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-20T07:39:00.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T07:39:00.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:33:48.103889', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhw/JKRnkaJADFLDR3Xa7VLMAIBk8pQK5DHw2bEuYRAlZcpIRX+cmrW8BQEotiKhl8n9TUmjY4AFghGAYsFqGY='} 2024-02-20T07:39:00.710 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:33:48.102433', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6cYsuhI8MozMWzH0Sjph1vRRLMXMVXpIMfcT2j0scvrS/+Bid6b3tOdWStzxUFp6jvMOSl44DdhGZSejjjL5s='} 2024-02-20T07:39:00.710 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-20T07:39:00.721 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-20T07:39:00.722 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-20T07:39:00.722 INFO:teuthology.run_tasks:Running task console_log... 2024-02-20T07:39:00.796 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0ff55250d0>, signals=[15]) 2024-02-20T07:39:00.797 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-20T07:39:00.803 INFO:teuthology.task.internal:Opening connections... 2024-02-20T07:39:00.803 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:39:00.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:00.912 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-02-20T07:39:00.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:01.001 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-20T07:39:01.008 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-02-20T07:39:01.025 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-02-20T07:39:01.026 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-02-20T07:39:01.082 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-02-20T07:39:01.082 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-02-20T07:39:01.082 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-02-20T07:39:01.082 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-02-20T07:39:01.082 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-02-20T07:39:01.082 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-20T07:39:01.083 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T07:39:01.084 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-02-20T07:39:01.106 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-02-20T07:39:01.125 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-02-20T07:39:01.125 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-20T07:39:01.182 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-02-20T07:39:01.183 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-02-20T07:39:01.183 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-20T07:39:01.183 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-02-20T07:39:01.183 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T07:39:01.183 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-20T07:39:01.183 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T07:39:01.183 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-02-20T07:39:01.206 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-20T07:39:01.215 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-20T07:39:01.220 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-20T07:39:01.220 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-02-20T07:39:01.223 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-02-20T07:39:01.238 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-20T07:39:01.243 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-20T07:39:01.244 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-02-20T07:39:01.280 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-02-20T07:39:01.376 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-20T07:39:01.437 INFO:teuthology.run_tasks:Running task kernel... 2024-02-20T07:39:01.452 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-20T07:39:01.452 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-20T07:39:01.453 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T07:39:01.453 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T07:39:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:39:01.471 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-02-20T07:39:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:39:01.473 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-02-20T07:39:01.527 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2024-02-20T07:39:01.527 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2024-02-20T07:39:01.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2024-02-20T07:39:01.530 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-02-20T07:39:01.530 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-02-20T07:39:01.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-02-20T07:39:03.640 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-20T07:39:03.738 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-20T07:39:05.041 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-02-20T07:39:05.188 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-02-20T07:39:14.263 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 906 kB/s | 8.0 MB 00:09 2024-02-20T07:39:15.144 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 851 kB/s | 8.0 MB 00:09 2024-02-20T07:39:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-02-20T07:39:16.979 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-02-20T07:39:20.873 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-02-20T07:39:22.082 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-02-20T07:39:22.291 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-02-20T07:39:23.472 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 15 kB 00:00 2024-02-20T07:39:23.550 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-02-20T07:39:24.865 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-02-20T07:39:28.442 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-02-20T07:39:29.924 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-02-20T07:39:30.749 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-20T07:39:30.749 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-20T07:39:30.854 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:39:30.861 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:39:30.861 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:39:30.861 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:39:30.861 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-02-20T07:39:30.862 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:39:30.863 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 100 M 2024-02-20T07:39:30.863 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-02-20T07:39:30.863 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:39:31.235 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 16 MB/s | 6.1 MB 00:00 2024-02-20T07:39:31.902 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-02-20T07:39:32.280 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-20T07:39:32.472 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-20T07:39:32.472 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:39:32.472 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:39:32.472 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 100 M 2024-02-20T07:39:32.473 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 123 M 2024-02-20T07:39:32.474 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:39:32.677 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 23 MB/s | 33 MB 00:01 2024-02-20T07:39:32.750 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 17 MB/s | 6.1 MB 00:00 2024-02-20T07:39:33.085 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-20T07:39:33.089 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:39:33.089 INFO:teuthology.orchestra.run.smithi186.stdout:Total 45 MB/s | 100 MB 00:02 2024-02-20T07:39:33.089 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:39:33.258 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-20T07:39:33.476 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:39:33.477 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:39:34.058 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 26 MB/s | 33 MB 00:01 2024-02-20T07:39:34.523 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:39:34.523 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:39:34.533 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-20T07:39:34.537 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:39:34.537 INFO:teuthology.orchestra.run.smithi055.stdout:Total 46 MB/s | 100 MB 00:02 2024-02-20T07:39:34.537 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:39:34.985 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:39:34.985 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:39:36.057 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:39:36.057 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:39:36.949 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:39:37.604 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:39:37.735 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:39:38.625 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:39:39.310 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:39:39.435 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:39:40.458 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:39:40.638 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:39:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:39:42.514 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:39:45.356 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:39:45.459 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:39:47.623 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:39:47.727 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:39:50.173 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:39:52.848 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:19.537 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:19.538 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-20T07:40:19.538 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:40:19.608 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:20.097 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:20.098 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:40:20.098 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:40:20.098 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:40:20.554 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:40:20.555 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:40:21.065 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:40:21.195 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:40:21.196 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:40:21.196 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-20T07:40:21.196 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T07:40:21.196 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T07:40:21.196 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-20T07:40:21.196 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T07:40:21.196 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:40:21.303 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:40:21.304 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:40:21.304 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-02-20T07:40:21.338 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-02-20T07:40:21.339 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-02-20T07:40:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-02-20T07:40:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-20T07:40:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-20T07:40:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-20T07:40:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-20T07:40:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T07:40:21.341 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T07:40:21.342 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T07:40:21.342 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-02-20T07:40:21.342 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:40:21.342 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T07:40:21.344 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T07:40:21.344 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-421.el9.x86_64 2024-02-20T07:40:21.344 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T07:40:21.967 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-02-20T07:40:23.367 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-20T07:40:23.404 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-02-20T07:40:23.406 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-02-20T07:40:23.421 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.RLLg5ddd5L 2024-02-20T07:40:23.421 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RLLg5ddd5L 2024-02-20T07:40:23.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.RLLg5ddd5L 2024-02-20T07:40:23.633 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.RLLg5ddd5L is 6KB 2024-02-20T07:40:23.682 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.RLLg5ddd5L 2024-02-20T07:40:23.696 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T07:40:23.727 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:23.728 INFO:teuthology.orchestra.run.smithi055.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-20T07:40:23.728 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:40:23.763 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-20T07:40:23.763 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-20T07:40:23.763 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-20T07:40:23.764 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-20T07:40:23.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-20T07:40:23.800 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:23.901 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-02-20T07:40:24.241 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:24.241 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:40:24.241 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:40:24.241 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:40:24.675 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:40:24.675 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:40:24.676 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:40:25.177 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:40:25.313 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:40:25.313 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:40:25.313 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-20T07:40:25.314 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T07:40:25.314 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T07:40:25.314 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-20T07:40:25.314 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T07:40:25.314 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:40:25.427 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:40:25.427 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:40:25.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2024-02-20T07:40:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T07:40:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T07:40:25.468 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2024-02-20T07:40:25.468 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:40:25.468 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T07:40:25.469 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T07:40:25.469 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-421.el9.x86_64 2024-02-20T07:40:25.469 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T07:40:26.104 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-02-20T07:40:27.470 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-20T07:40:27.508 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-02-20T07:40:27.510 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-02-20T07:40:27.525 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.FInFPdQ2Hs 2024-02-20T07:40:27.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FInFPdQ2Hs 2024-02-20T07:40:27.600 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.FInFPdQ2Hs 2024-02-20T07:40:27.740 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.FInFPdQ2Hs is 6KB 2024-02-20T07:40:27.790 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.FInFPdQ2Hs 2024-02-20T07:40:27.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T07:40:27.873 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-20T07:40:27.873 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-20T07:40:27.873 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-20T07:40:27.873 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-20T07:40:27.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-20T07:40:28.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-02-20T07:40:53.912 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-20T07:40:53.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:40:53.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:40:58.036 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-20T07:40:58.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-02-20T07:40:58.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:41:12.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T07:41:16.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-02-20T07:41:21.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:41:21.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:41:24.448 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T07:41:25.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-02-20T07:41:25.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:41:36.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:41:36.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:41:54.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T07:42:09.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:42:09.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:42:10.238 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-02-20T07:42:10.785 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:42:10.785 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-421.el9.x86_64"... 2024-02-20T07:42:10.786 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-02-20T07:42:10.799 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-421.el9.x86_64 2024-02-20T07:42:10.800 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-20T07:42:10.800 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T07:42:10.800 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T07:42:11.803 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-20T07:42:11.803 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T07:42:11.889 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-02-20T07:42:11.924 DEBUG:teuthology.parallel:result is None 2024-02-20T07:42:25.482 DEBUG:teuthology.orchestra.remote:timed out 2024-02-20T07:42:37.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-02-20T07:42:37.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:42:37.770 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-02-20T07:42:38.171 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-02-20T07:42:38.171 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-421.el9.x86_64"... 2024-02-20T07:42:38.171 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-02-20T07:42:38.187 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-421.el9.x86_64 2024-02-20T07:42:38.187 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-20T07:42:38.187 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T07:42:38.187 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T07:42:39.188 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-20T07:42:39.188 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T07:42:39.266 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-02-20T07:42:39.306 DEBUG:teuthology.parallel:result is None 2024-02-20T07:42:39.306 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-20T07:42:39.318 INFO:teuthology.task.internal:Creating test directory... 2024-02-20T07:42:39.319 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T07:42:39.321 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T07:42:39.340 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-20T07:42:39.377 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-20T07:42:39.383 INFO:teuthology.task.internal:Creating archive directory... 2024-02-20T07:42:39.384 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T07:42:39.386 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T07:42:39.462 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-20T07:42:39.468 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-20T07:42:39.469 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T07:42:39.471 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T07:42:39.504 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:42:39.508 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:42:39.519 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:42:39.523 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:42:39.524 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-20T07:42:39.531 INFO:teuthology.task.internal:Configuring sudo... 2024-02-20T07:42:39.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T07:42:39.567 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T07:42:39.598 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-20T07:42:39.607 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-20T07:42:39.608 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T07:42:39.638 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T07:42:39.655 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:42:39.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:42:39.795 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:42:39.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:42:39.922 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:42:39.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T07:42:39.990 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:42:40.031 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:42:40.098 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:42:40.141 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:42:40.188 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:42:40.188 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T07:42:40.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-02-20T07:42:40.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-02-20T07:42:40.303 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T07:42:40.357 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T07:42:40.758 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-20T07:42:40.766 INFO:teuthology.task.internal:Starting timer... 2024-02-20T07:42:40.766 INFO:teuthology.run_tasks:Running task pcp... 2024-02-20T07:42:40.776 INFO:teuthology.run_tasks:Running task selinux... 2024-02-20T07:42:40.785 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-20T07:42:40.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-02-20T07:42:40.847 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-02-20T07:42:40.850 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-02-20T07:42:40.915 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-02-20T07:42:40.916 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-20T07:42:40.916 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-02-20T07:42:40.941 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-02-20T07:42:40.942 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-02-20T07:42:40.967 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-02-20T07:42:40.967 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-02-20T07:42:40.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T07:42:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:42:41.011 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T07:42:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:42:41.038 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-20T07:42:41.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-02-20T07:42:41.085 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-02-20T07:42:41.119 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-20T07:42:41.128 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-20T07:42:41.133 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-20T07:42:41.261 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-20T07:42:41.281 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-20T07:42:41.282 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-20T07:46:57.069 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-02-20T07:46:57.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-02-20T07:46:57.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:46:57.165 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-02-20T07:46:57.247 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-02-20T07:46:57.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:46:57.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:46:57.336 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-02-20T07:46:57.423 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T07:46:57.424 INFO:teuthology.run_tasks:Running task clock... 2024-02-20T07:46:57.435 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-20T07:46:57.436 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T07:46:57.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T07:46:57.439 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T07:46:57.439 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T07:46:57.468 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T07:46:57.485 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T07:46:57.497 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T07:46:57.514 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T07:46:57.515 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-02-20T07:46:57.529 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-02-20T07:46:57.545 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-02-20T07:46:57.546 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T07:46:57.558 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-02-20T07:46:57.562 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T07:46:57.574 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T07:46:57.591 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T07:46:57.616 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-02-20T07:46:57.620 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T07:46:57.620 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-02-20T07:46:57.620 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.620 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.620 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.620 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.639 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-02-20T07:46:57.642 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T07:46:57.642 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-02-20T07:46:57.642 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.642 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.643 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.643 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:46:57.643 INFO:teuthology.run_tasks:Running task install... 2024-02-20T07:46:57.651 DEBUG:teuthology.task.install:project ceph 2024-02-20T07:46:57.652 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c'}} 2024-02-20T07:46:57.652 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c'} 2024-02-20T07:46:57.652 INFO:teuthology.task.install:Using flavor: default 2024-02-20T07:46:57.659 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-20T07:46:57.659 INFO:teuthology.task.install:extra packages: [] 2024-02-20T07:46:57.660 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c', 'tag': None, 'wait_for_package': False} 2024-02-20T07:46:57.660 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:46:57.660 INFO:teuthology.packaging:ref: None 2024-02-20T07:46:57.660 INFO:teuthology.packaging:tag: None 2024-02-20T07:46:57.660 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:46:57.661 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:46:57.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:46:57.663 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c', 'tag': None, 'wait_for_package': False} 2024-02-20T07:46:57.663 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:46:57.663 INFO:teuthology.packaging:ref: None 2024-02-20T07:46:57.663 INFO:teuthology.packaging:tag: None 2024-02-20T07:46:57.663 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:46:57.663 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:46:57.663 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:46:57.916 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/ 2024-02-20T07:46:57.917 INFO:teuthology.task.install.rpm:Package version is 17.2.7-521.gc80e7168 2024-02-20T07:46:57.924 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/ 2024-02-20T07:46:57.924 INFO:teuthology.task.install.rpm:Package version is 17.2.7-521.gc80e7168 2024-02-20T07:46:58.149 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T07:46:58.149 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:46:58.149 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T07:46:58.179 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T07:46:58.179 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:46:58.179 INFO:teuthology.packaging:ref: None 2024-02-20T07:46:58.179 INFO:teuthology.packaging:tag: None 2024-02-20T07:46:58.180 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:46:58.180 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:46:58.180 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T07:46:58.215 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T07:46:58.216 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:46:58.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T07:46:58.244 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T07:46:58.245 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:46:58.245 INFO:teuthology.packaging:ref: None 2024-02-20T07:46:58.245 INFO:teuthology.packaging:tag: None 2024-02-20T07:46:58.245 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:46:58.245 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:46:58.245 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T07:46:58.249 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T07:46:58.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T07:46:58.329 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T07:46:58.393 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T07:46:58.396 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-02-20T07:46:58.397 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-02-20T07:46:58.460 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2024-02-20T07:46:58.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2024-02-20T07:46:58.654 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-02-20T07:46:58.684 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw 2024-02-20T07:46:58.713 INFO:teuthology.orchestra.run.smithi055.stdout:74 files removed 2024-02-20T07:46:58.740 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2024-02-20T07:46:59.565 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 138 kB/s | 77 kB 00:00 2024-02-20T07:46:59.598 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 144 kB/s | 77 kB 00:00 2024-02-20T07:46:59.924 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-02-20T07:46:59.966 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-02-20T07:47:00.274 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-02-20T07:47:00.299 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 6.0 kB/s | 1.8 kB 00:00 2024-02-20T07:47:00.726 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-02-20T07:47:00.885 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-20T07:47:02.289 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-02-20T07:47:02.369 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-02-20T07:47:06.168 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-02-20T07:47:11.801 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 897 kB/s | 8.0 MB 00:09 2024-02-20T07:47:13.185 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 19 MB 00:05 2024-02-20T07:47:18.645 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.1 MB 00:01 2024-02-20T07:47:20.429 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-02-20T07:47:22.645 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-02-20T07:47:23.680 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 20 MB 00:03 2024-02-20T07:47:26.976 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.1 MB 00:00 2024-02-20T07:47:28.627 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-02-20T07:47:28.764 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-02-20T07:47:31.340 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 20 MB 00:02 2024-02-20T07:47:31.483 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:47:31.493 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:47:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:47:31.493 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:47:31.493 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:47:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 14 M 2024-02-20T07:47:31.493 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.1 M 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:17.2.7-521.gc80e7168.el9 ceph 5.0 M 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 23 M 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 635 k 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-20T07:47:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 457 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 4.9 M 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 45 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 116 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:17.2.7-521.gc80e7168.el9 ceph 156 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:17.2.7-521.gc80e7168.el9 ceph 320 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 293 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 99 k 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-20T07:47:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Install 22 Packages 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 58 M 2024-02-20T07:47:31.496 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:47:32.115 INFO:teuthology.orchestra.run.smithi055.stdout:(1/24): ceph-base-17.2.7-521.gc80e7168.el9.x86_ 8.8 MB/s | 5.0 MB 00:00 2024-02-20T07:47:32.182 INFO:teuthology.orchestra.run.smithi055.stdout:(2/24): ceph-selinux-17.2.7-521.gc80e7168.el9.x 377 kB/s | 25 kB 00:00 2024-02-20T07:47:32.266 INFO:teuthology.orchestra.run.smithi055.stdout:(3/24): libcephfs2-17.2.7-521.gc80e7168.el9.x86 7.5 MB/s | 635 kB 00:00 2024-02-20T07:47:32.349 INFO:teuthology.orchestra.run.smithi055.stdout:(4/24): libradosstriper1-17.2.7-521.gc80e7168.e 5.4 MB/s | 457 kB 00:00 2024-02-20T07:47:32.733 INFO:teuthology.orchestra.run.smithi055.stdout:(5/24): ceph-radosgw-17.2.7-521.gc80e7168.el9.x 12 MB/s | 14 MB 00:01 2024-02-20T07:47:32.808 INFO:teuthology.orchestra.run.smithi055.stdout:(6/24): librgw2-17.2.7-521.gc80e7168.el9.x86_64 11 MB/s | 4.9 MB 00:00 2024-02-20T07:47:32.834 INFO:teuthology.orchestra.run.smithi055.stdout:(7/24): python3-ceph-argparse-17.2.7-521.gc80e7 450 kB/s | 45 kB 00:00 2024-02-20T07:47:32.859 INFO:teuthology.orchestra.run.smithi055.stdout:(8/24): python3-ceph-common-17.2.7-521.gc80e716 2.3 MB/s | 116 kB 00:00 2024-02-20T07:47:32.885 INFO:teuthology.orchestra.run.smithi055.stdout:(9/24): python3-cephfs-17.2.7-521.gc80e7168.el9 3.0 MB/s | 156 kB 00:00 2024-02-20T07:47:32.910 INFO:teuthology.orchestra.run.smithi055.stdout:(10/24): python3-rados-17.2.7-521.gc80e7168.el9 6.2 MB/s | 320 kB 00:00 2024-02-20T07:47:32.935 INFO:teuthology.orchestra.run.smithi055.stdout:(11/24): python3-rbd-17.2.7-521.gc80e7168.el9.x 5.7 MB/s | 293 kB 00:00 2024-02-20T07:47:32.960 INFO:teuthology.orchestra.run.smithi055.stdout:(12/24): python3-rgw-17.2.7-521.gc80e7168.el9.x 1.9 MB/s | 99 kB 00:00 2024-02-20T07:47:33.169 INFO:teuthology.orchestra.run.smithi055.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 212 kB/s | 44 kB 00:00 2024-02-20T07:47:33.211 INFO:teuthology.orchestra.run.smithi055.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 578 kB/s | 159 kB 00:00 2024-02-20T07:47:33.311 INFO:teuthology.orchestra.run.smithi055.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-02-20T07:47:33.337 INFO:teuthology.orchestra.run.smithi055.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-02-20T07:47:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-02-20T07:47:33.554 INFO:teuthology.orchestra.run.smithi055.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-02-20T07:47:33.621 INFO:teuthology.orchestra.run.smithi055.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-02-20T07:47:33.663 INFO:teuthology.orchestra.run.smithi055.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-02-20T07:47:33.880 INFO:teuthology.orchestra.run.smithi055.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-02-20T07:47:33.947 INFO:teuthology.orchestra.run.smithi055.stdout:(22/24): librados2-17.2.7-521.gc80e7168.el9.x86 11 MB/s | 3.1 MB 00:00 2024-02-20T07:47:34.172 INFO:teuthology.orchestra.run.smithi055.stdout:(23/24): librbd1-17.2.7-521.gc80e7168.el9.x86_6 10 MB/s | 2.9 MB 00:00 2024-02-20T07:47:36.295 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-02-20T07:47:38.916 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:47:38.924 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:47:38.924 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 14 M 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.1 M 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:17.2.7-521.gc80e7168.el9 ceph 5.0 M 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 23 M 2024-02-20T07:47:38.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 635 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 457 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 4.9 M 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 45 k 2024-02-20T07:47:38.927 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 116 k 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:17.2.7-521.gc80e7168.el9 ceph 156 k 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:17.2.7-521.gc80e7168.el9 ceph 320 k 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 293 k 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 99 k 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout:Install 22 Packages 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 2 Packages 2024-02-20T07:47:38.928 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:47:38.929 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 58 M 2024-02-20T07:47:38.929 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:47:39.763 INFO:teuthology.orchestra.run.smithi186.stdout:(1/24): ceph-base-17.2.7-521.gc80e7168.el9.x86_ 6.4 MB/s | 5.0 MB 00:00 2024-02-20T07:47:39.863 INFO:teuthology.orchestra.run.smithi186.stdout:(2/24): ceph-selinux-17.2.7-521.gc80e7168.el9.x 252 kB/s | 25 kB 00:00 2024-02-20T07:47:39.972 INFO:teuthology.orchestra.run.smithi186.stdout:(3/24): libcephfs2-17.2.7-521.gc80e7168.el9.x86 5.7 MB/s | 635 kB 00:00 2024-02-20T07:47:40.080 INFO:teuthology.orchestra.run.smithi186.stdout:(4/24): libradosstriper1-17.2.7-521.gc80e7168.e 4.1 MB/s | 457 kB 00:00 2024-02-20T07:47:40.631 INFO:teuthology.orchestra.run.smithi186.stdout:(5/24): librgw2-17.2.7-521.gc80e7168.el9.x86_64 8.9 MB/s | 4.9 MB 00:00 2024-02-20T07:47:40.731 INFO:teuthology.orchestra.run.smithi186.stdout:(6/24): python3-ceph-argparse-17.2.7-521.gc80e7 452 kB/s | 45 kB 00:00 2024-02-20T07:47:40.831 INFO:teuthology.orchestra.run.smithi186.stdout:(7/24): python3-ceph-common-17.2.7-521.gc80e716 1.1 MB/s | 116 kB 00:00 2024-02-20T07:47:40.923 INFO:teuthology.orchestra.run.smithi186.stdout:(8/24): python3-cephfs-17.2.7-521.gc80e7168.el9 1.7 MB/s | 156 kB 00:00 2024-02-20T07:47:41.024 INFO:teuthology.orchestra.run.smithi186.stdout:(9/24): python3-rados-17.2.7-521.gc80e7168.el9. 3.1 MB/s | 320 kB 00:00 2024-02-20T07:47:41.199 INFO:teuthology.orchestra.run.smithi186.stdout:(10/24): ceph-radosgw-17.2.7-521.gc80e7168.el9. 6.3 MB/s | 14 MB 00:02 2024-02-20T07:47:41.224 INFO:teuthology.orchestra.run.smithi186.stdout:(11/24): python3-rbd-17.2.7-521.gc80e7168.el9.x 1.4 MB/s | 293 kB 00:00 2024-02-20T07:47:41.283 INFO:teuthology.orchestra.run.smithi186.stdout:(12/24): python3-rgw-17.2.7-521.gc80e7168.el9.x 1.2 MB/s | 99 kB 00:00 2024-02-20T07:47:41.516 INFO:teuthology.orchestra.run.smithi186.stdout:(13/24): ceph-common-17.2.7-521.gc80e7168.el9.x 8.9 MB/s | 23 MB 00:02 2024-02-20T07:47:41.558 INFO:teuthology.orchestra.run.smithi186.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 477 kB/s | 159 kB 00:00 2024-02-20T07:47:41.583 INFO:teuthology.orchestra.run.smithi186.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 147 kB/s | 44 kB 00:00 2024-02-20T07:47:41.667 INFO:teuthology.orchestra.run.smithi186.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-02-20T07:47:41.693 INFO:teuthology.orchestra.run.smithi186.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-02-20T07:47:41.718 INFO:teuthology.orchestra.run.smithi186.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 308 kB/s | 41 kB 00:00 2024-02-20T07:47:41.818 INFO:teuthology.orchestra.run.smithi186.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-02-20T07:47:41.852 INFO:teuthology.orchestra.run.smithi186.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-02-20T07:47:41.944 INFO:teuthology.orchestra.run.smithi186.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-02-20T07:47:42.294 INFO:teuthology.orchestra.run.smithi186.stdout:(22/24): librados2-17.2.7-521.gc80e7168.el9.x86 7.0 MB/s | 3.1 MB 00:00 2024-02-20T07:47:42.352 INFO:teuthology.orchestra.run.smithi186.stdout:(23/24): librbd1-17.2.7-521.gc80e7168.el9.x86_6 7.2 MB/s | 2.9 MB 00:00 2024-02-20T07:47:42.553 INFO:teuthology.orchestra.run.smithi186.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-02-20T07:47:42.556 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:42.556 INFO:teuthology.orchestra.run.smithi186.stdout:Total 16 MB/s | 58 MB 00:03 2024-02-20T07:47:42.557 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:47:42.590 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:47:42.590 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:47:42.856 INFO:teuthology.orchestra.run.smithi055.stdout:(24/24): ceph-common-17.2.7-521.gc80e7168.el9.x 2.0 MB/s | 23 MB 00:11 2024-02-20T07:47:42.860 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:42.860 INFO:teuthology.orchestra.run.smithi055.stdout:Total 5.1 MB/s | 58 MB 00:11 2024-02-20T07:47:42.861 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:47:42.898 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:47:42.898 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:47:43.051 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:47:43.051 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:47:43.368 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:47:43.368 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:47:44.286 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:47:44.629 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:47:44.666 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-20T07:47:44.696 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:47:44.771 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:47:44.850 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:47:44.898 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:47:44.993 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-20T07:47:44.995 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:47:45.040 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-20T07:47:45.041 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:47:45.069 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-20T07:47:45.134 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:47:45.214 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:47:45.252 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:47:45.324 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:47:45.369 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-20T07:47:45.398 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-20T07:47:45.504 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-20T07:47:45.534 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:47:45.597 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:47:45.625 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-20T07:47:45.669 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 10/26 2024-02-20T07:47:45.708 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:47:45.776 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:47:45.803 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:47:45.852 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:47:45.855 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-20T07:47:45.901 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-20T07:47:45.913 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:47:45.945 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-20T07:47:45.977 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-20T07:47:45.984 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:47:46.021 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-20T07:47:46.066 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 10/26 2024-02-20T07:47:46.102 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:47:46.166 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:47:46.192 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:47:46.257 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-20T07:47:46.297 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-20T07:47:46.316 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:47:46.334 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-20T07:47:46.374 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-20T07:47:46.405 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:47:46.478 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 19/26 2024-02-20T07:47:46.510 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 20/26 2024-02-20T07:47:46.695 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-20T07:47:46.721 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:47:46.812 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:47:46.913 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 19/26 2024-02-20T07:47:46.941 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 20/26 2024-02-20T07:47:48.928 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:47:48.973 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:47:49.477 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:47:49.509 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:47:49.512 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:47:49.551 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:47:49.628 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:47:49.628 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-20T07:47:49.628 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-20T07:47:49.628 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:47:49.661 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:47:50.070 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:47:50.108 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:47:50.198 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:47:50.198 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-20T07:47:50.198 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-20T07:47:50.198 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:47:50.235 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-02-20T07:47:59.251 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:00.172 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:48:00.172 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2024-02-20T07:48:00.172 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2024-02-20T07:48:00.172 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2024-02-20T07:48:00.172 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2024-02-20T07:48:00.172 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2024-02-20T07:48:00.173 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2024-02-20T07:48:00.173 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2024-02-20T07:48:00.173 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:00.543 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:48:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:48:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-20T07:48:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:48:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:48:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:00.607 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:48:00.705 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:48:00.705 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:48:01.486 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:48:01.553 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:48:01.553 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:01.553 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-20T07:48:01.553 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:48:01.553 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:48:01.553 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:01.555 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:48:01.631 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:48:01.631 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:48:01.967 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:48:01.967 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 1/26 2024-02-20T07:48:01.967 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:48:01.967 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 5/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 6/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 7/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 8/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 9/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 10/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-20T07:48:01.968 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-20T07:48:01.969 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-20T07:48:01.969 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-20T07:48:01.969 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-20T07:48:01.970 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-20T07:48:01.970 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-20T07:48:01.970 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-20T07:48:01.970 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:48:01.970 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-20T07:48:01.970 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 25/26 2024-02-20T07:48:02.874 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:48:02.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 1/26 2024-02-20T07:48:02.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:48:02.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:48:02.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:48:02.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 5/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 6/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 7/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 8/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 9/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 10/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-20T07:48:02.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-20T07:48:02.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-20T07:48:02.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-20T07:48:02.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-20T07:48:02.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:48:02.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-20T07:48:02.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 25/26 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.885 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:02.887 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-20T07:48:02.887 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:02.887 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:03.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test 2024-02-20T07:48:03.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:48:03.685 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:03.685 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.686 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:03.687 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:03.821 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:27 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:03.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2024-02-20T07:48:03.989 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:03.989 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:03.989 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:17.2.7-521.gc80e7168.el9 ceph 48 M 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:17.2.7-521.gc80e7168.el9 ceph 163 k 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:03.990 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-02-20T07:48:03.991 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:03.991 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 49 M 2024-02-20T07:48:03.991 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 196 M 2024-02-20T07:48:03.991 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:48:04.318 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 919 kB/s | 300 kB 00:00 2024-02-20T07:48:04.385 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 938 kB/s | 63 kB 00:00 2024-02-20T07:48:04.486 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 186 kB 00:00 2024-02-20T07:48:04.558 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:36 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:04.578 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-02-20T07:48:04.603 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): libcephsqlite-17.2.7-521.gc80e7168.el9.x 266 kB/s | 163 kB 00:00 2024-02-20T07:48:04.732 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:17.2.7-521.gc80e7168.el9 ceph 48 M 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-20T07:48:04.733 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:17.2.7-521.gc80e7168.el9 ceph 163 k 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-02-20T07:48:04.734 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:04.735 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 49 M 2024-02-20T07:48:04.735 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 196 M 2024-02-20T07:48:04.735 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:04.994 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): libcephsqlite-17.2.7-521.gc80e7168.el9.x 630 kB/s | 163 kB 00:00 2024-02-20T07:48:05.145 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 734 kB/s | 300 kB 00:00 2024-02-20T07:48:05.170 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 357 kB/s | 63 kB 00:00 2024-02-20T07:48:05.295 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2024-02-20T07:48:05.362 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-02-20T07:48:07.112 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-test-17.2.7-521.gc80e7168.el9.x86_6 20 MB/s | 48 MB 00:02 2024-02-20T07:48:07.115 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:07.116 INFO:teuthology.orchestra.run.smithi055.stdout:Total 21 MB/s | 49 MB 00:02 2024-02-20T07:48:07.116 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:07.137 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:07.137 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:07.543 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:07.544 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:08.153 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:08.243 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-20T07:48:08.285 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-20T07:48:08.329 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-20T07:48:08.377 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-20T07:48:08.405 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:48:13.055 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:48:13.140 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:48:15.496 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:48:15.496 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 1/6 2024-02-20T07:48:15.496 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2/6 2024-02-20T07:48:15.496 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-20T07:48:15.496 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-20T07:48:15.497 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-20T07:48:15.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-20T07:48:15.875 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:15.875 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:15.875 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:15.876 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2024-02-20T07:48:15.876 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:15.876 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-20T07:48:15.876 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-20T07:48:15.876 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-20T07:48:15.877 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:15.877 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:16.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2024-02-20T07:48:16.733 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:48 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:16.910 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:16.912 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-02-20T07:48:16.912 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:16.912 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-02-20T07:48:16.912 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:17.2.7-521.gc80e7168.el9 ceph 6.5 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.1 M 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:17.2.7-521.gc80e7168.el9 ceph 1.4 M 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 239 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.7 M 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 15 M 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-20T07:48:16.913 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-20T07:48:16.914 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout:Install 33 Packages 2024-02-20T07:48:16.915 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:16.916 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 27 M 2024-02-20T07:48:16.916 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 99 M 2024-02-20T07:48:16.916 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:17.495 INFO:teuthology.orchestra.run.smithi055.stdout:(1/33): ceph-17.2.7-521.gc80e7168.el9.x86_64.rp 29 kB/s | 6.5 kB 00:00 2024-02-20T07:48:18.629 INFO:teuthology.orchestra.run.smithi055.stdout:(2/33): ceph-mgr-17.2.7-521.gc80e7168.el9.x86_6 1.0 MB/s | 1.4 MB 00:01 2024-02-20T07:48:18.879 INFO:teuthology.orchestra.run.smithi055.stdout:(3/33): ceph-mon-17.2.7-521.gc80e7168.el9.x86_6 2.7 MB/s | 3.7 MB 00:01 2024-02-20T07:48:18.946 INFO:teuthology.orchestra.run.smithi055.stdout:(4/33): ceph-mgr-modules-core-17.2.7-521.gc80e7 3.5 MB/s | 239 kB 00:00 2024-02-20T07:48:19.055 INFO:teuthology.orchestra.run.smithi055.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-02-20T07:48:19.114 INFO:teuthology.orchestra.run.smithi055.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 541 kB/s | 32 kB 00:00 2024-02-20T07:48:19.156 INFO:teuthology.orchestra.run.smithi055.stdout:(7/33): ceph-mds-17.2.7-521.gc80e7168.el9.x86_6 1.1 MB/s | 2.1 MB 00:01 2024-02-20T07:48:19.181 INFO:teuthology.orchestra.run.smithi055.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 709 kB/s | 47 kB 00:00 2024-02-20T07:48:19.206 INFO:teuthology.orchestra.run.smithi055.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-02-20T07:48:19.248 INFO:teuthology.orchestra.run.smithi055.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-02-20T07:48:19.315 INFO:teuthology.orchestra.run.smithi055.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-02-20T07:48:19.340 INFO:teuthology.orchestra.run.smithi055.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-02-20T07:48:19.374 INFO:teuthology.orchestra.run.smithi055.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-02-20T07:48:19.399 INFO:teuthology.orchestra.run.smithi055.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 1.9 MB/s | 113 kB 00:00 2024-02-20T07:48:19.424 INFO:teuthology.orchestra.run.smithi055.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-20T07:48:19.733 INFO:teuthology.orchestra.run.smithi055.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 130 kB/s | 43 kB 00:00 2024-02-20T07:48:19.833 INFO:teuthology.orchestra.run.smithi055.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 420 kB/s | 172 kB 00:00 2024-02-20T07:48:19.942 INFO:teuthology.orchestra.run.smithi055.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-02-20T07:48:19.976 INFO:teuthology.orchestra.run.smithi055.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-02-20T07:48:20.042 INFO:teuthology.orchestra.run.smithi055.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-02-20T07:48:20.068 INFO:teuthology.orchestra.run.smithi055.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-02-20T07:48:20.143 INFO:teuthology.orchestra.run.smithi055.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-02-20T07:48:20.168 INFO:teuthology.orchestra.run.smithi055.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-02-20T07:48:20.293 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): ceph-test-17.2.7-521.gc80e7168.el9.x86_6 2.9 MB/s | 48 MB 00:16 2024-02-20T07:48:20.296 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:20.297 INFO:teuthology.orchestra.run.smithi186.stdout:Total 3.0 MB/s | 49 MB 00:16 2024-02-20T07:48:20.297 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:48:20.316 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:48:20.319 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:48:20.369 INFO:teuthology.orchestra.run.smithi055.stdout:(24/33): ceph-osd-17.2.7-521.gc80e7168.el9.x86_ 8.8 MB/s | 15 MB 00:01 2024-02-20T07:48:20.394 INFO:teuthology.orchestra.run.smithi055.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 185 kB/s | 46 kB 00:00 2024-02-20T07:48:20.419 INFO:teuthology.orchestra.run.smithi055.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 315 kB/s | 79 kB 00:00 2024-02-20T07:48:20.494 INFO:teuthology.orchestra.run.smithi055.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2024-02-20T07:48:20.519 INFO:teuthology.orchestra.run.smithi055.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 896 kB/s | 90 kB 00:00 2024-02-20T07:48:20.603 INFO:teuthology.orchestra.run.smithi055.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-02-20T07:48:20.628 INFO:teuthology.orchestra.run.smithi055.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-02-20T07:48:20.728 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:48:20.728 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:48:20.737 INFO:teuthology.orchestra.run.smithi055.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-02-20T07:48:20.796 INFO:teuthology.orchestra.run.smithi055.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2024-02-20T07:48:20.862 INFO:teuthology.orchestra.run.smithi055.stdout:(33/33): python3-pecan-1.4.2-3.el9.noarch.rpm 552 kB/s | 272 kB 00:00 2024-02-20T07:48:20.866 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:20.866 INFO:teuthology.orchestra.run.smithi055.stdout:Total 6.8 MB/s | 27 MB 00:03 2024-02-20T07:48:20.866 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:20.909 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:20.909 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:21.268 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:21.268 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:48:21.405 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-20T07:48:21.434 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-20T07:48:21.495 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-20T07:48:21.535 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-20T07:48:21.555 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:48:22.552 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:22.624 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-20T07:48:22.657 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-20T07:48:22.698 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-20T07:48:22.793 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-20T07:48:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-20T07:48:22.926 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-20T07:48:22.999 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-20T07:48:23.069 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-20T07:48:23.106 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-20T07:48:23.164 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-20T07:48:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-20T07:48:23.410 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-20T07:48:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-20T07:48:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-20T07:48:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-20T07:48:23.572 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-20T07:48:23.650 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-20T07:48:23.691 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:48:23.740 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-20T07:48:23.781 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-20T07:48:23.816 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-20T07:48:23.850 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-20T07:48:23.942 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-20T07:48:23.999 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-20T07:48:24.090 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-20T07:48:24.228 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-20T07:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:48:24.513 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 28/33 2024-02-20T07:48:24.661 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:48:24.686 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:48:24.687 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:24.687 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-20T07:48:24.687 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:48:24.687 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:48:24.687 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:25.978 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:48:26.053 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:48:26.095 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:48:26.121 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:48:26.121 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:26.121 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-20T07:48:26.121 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:48:26.122 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:48:26.122 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:26.512 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:48:26.535 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:48:26.535 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:26.535 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-20T07:48:26.535 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:48:26.536 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:48:26.536 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:26.746 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:48:26.769 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:48:26.769 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:26.769 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-20T07:48:26.769 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:48:26.770 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:48:26.770 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:26.846 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:48:28.222 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:48:28.222 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 1/6 2024-02-20T07:48:28.223 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2/6 2024-02-20T07:48:28.223 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-20T07:48:28.223 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-20T07:48:28.223 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 1/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 3/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 4/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 5/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 6/33 2024-02-20T07:48:28.577 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-20T07:48:28.578 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-20T07:48:28.579 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-20T07:48:28.579 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-20T07:48:28.579 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-20T07:48:28.579 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-20T07:48:28.580 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:28.656 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-20T07:48:28.657 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-20T07:48:28.657 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-20T07:48:28.657 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:28.657 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:28.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:29.497 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-20T07:48:29.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-20T07:48:29.499 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:29.500 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:29.552 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:53 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:29.722 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:29.724 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-02-20T07:48:29.724 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:48:29.724 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:17.2.7-521.gc80e7168.el9 ceph 6.5 k 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.1 M 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:17.2.7-521.gc80e7168.el9 ceph 1.4 M 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 239 k 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.7 M 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 15 M 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-20T07:48:29.725 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-20T07:48:29.726 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout:Install 33 Packages 2024-02-20T07:48:29.727 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:29.728 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 27 M 2024-02-20T07:48:29.731 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 99 M 2024-02-20T07:48:29.732 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:48:29.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2024-02-20T07:48:30.353 INFO:teuthology.orchestra.run.smithi186.stdout:(1/33): ceph-17.2.7-521.gc80e7168.el9.x86_64.rp 20 kB/s | 6.5 kB 00:00 2024-02-20T07:48:30.403 INFO:teuthology.orchestra.run.smithi186.stdout:(2/33): ceph-mds-17.2.7-521.gc80e7168.el9.x86_6 5.5 MB/s | 2.1 MB 00:00 2024-02-20T07:48:30.429 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:02 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:30.546 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:48:30.602 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:30.603 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:48:30.603 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:30.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2024-02-20T07:48:31.104 INFO:teuthology.orchestra.run.smithi186.stdout:(3/33): ceph-osd-17.2.7-521.gc80e7168.el9.x86_6 22 MB/s | 15 MB 00:00 2024-02-20T07:48:31.154 INFO:teuthology.orchestra.run.smithi186.stdout:(4/33): ceph-mgr-modules-core-17.2.7-521.gc80e7 4.7 MB/s | 239 kB 00:00 2024-02-20T07:48:31.253 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:03 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:31.426 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:31.427 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:31.427 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:31.427 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:31.427 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:31.427 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 80 k 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 80 k 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 364 k 2024-02-20T07:48:31.428 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:31.454 INFO:teuthology.orchestra.run.smithi186.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 480 kB/s | 144 kB 00:00 2024-02-20T07:48:31.529 INFO:teuthology.orchestra.run.smithi186.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 421 kB/s | 32 kB 00:00 2024-02-20T07:48:31.630 INFO:teuthology.orchestra.run.smithi186.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 473 kB/s | 47 kB 00:00 2024-02-20T07:48:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 462 kB/s | 39 kB 00:00 2024-02-20T07:48:31.872 INFO:teuthology.orchestra.run.smithi186.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 241 kB 00:00 2024-02-20T07:48:31.986 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-17.2.7-521.gc80e7168.el9.noarch.rpm 144 kB/s | 80 kB 00:00 2024-02-20T07:48:31.987 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:31.987 INFO:teuthology.orchestra.run.smithi055.stdout:Total 144 kB/s | 80 kB 00:00 2024-02-20T07:48:31.987 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:31.993 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:31.993 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:31.995 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:31.995 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:32.039 INFO:teuthology.orchestra.run.smithi186.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 6.9 MB/s | 1.1 MB 00:00 2024-02-20T07:48:32.079 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:32.106 INFO:teuthology.orchestra.run.smithi186.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-20T07:48:32.189 INFO:teuthology.orchestra.run.smithi186.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-02-20T07:48:32.265 INFO:teuthology.orchestra.run.smithi186.stdout:(13/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 113 kB 00:00 2024-02-20T07:48:32.342 INFO:teuthology.orchestra.run.smithi186.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-02-20T07:48:32.409 INFO:teuthology.orchestra.run.smithi186.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 652 kB/s | 43 kB 00:00 2024-02-20T07:48:32.468 INFO:teuthology.orchestra.run.smithi186.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-02-20T07:48:32.518 INFO:teuthology.orchestra.run.smithi186.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-02-20T07:48:32.529 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:32.552 INFO:teuthology.orchestra.run.smithi186.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 322 kB/s | 11 kB 00:00 2024-02-20T07:48:32.585 INFO:teuthology.orchestra.run.smithi186.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 529 kB/s | 18 kB 00:00 2024-02-20T07:48:32.619 INFO:teuthology.orchestra.run.smithi186.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 697 kB/s | 23 kB 00:00 2024-02-20T07:48:32.628 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:32.661 INFO:teuthology.orchestra.run.smithi186.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-02-20T07:48:32.694 INFO:teuthology.orchestra.run.smithi186.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 594 kB/s | 20 kB 00:00 2024-02-20T07:48:32.728 INFO:teuthology.orchestra.run.smithi186.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-02-20T07:48:32.762 INFO:teuthology.orchestra.run.smithi186.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 2.3 MB/s | 79 kB 00:00 2024-02-20T07:48:32.812 INFO:teuthology.orchestra.run.smithi186.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-02-20T07:48:32.846 INFO:teuthology.orchestra.run.smithi186.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 492 kB/s | 16 kB 00:00 2024-02-20T07:48:32.879 INFO:teuthology.orchestra.run.smithi186.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.7 MB/s | 90 kB 00:00 2024-02-20T07:48:32.913 INFO:teuthology.orchestra.run.smithi186.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-02-20T07:48:32.947 INFO:teuthology.orchestra.run.smithi186.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-02-20T07:48:32.983 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:33.077 INFO:teuthology.orchestra.run.smithi186.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-02-20T07:48:33.077 INFO:teuthology.orchestra.run.smithi186.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 602 kB/s | 20 kB 00:00 2024-02-20T07:48:33.247 INFO:teuthology.orchestra.run.smithi186.stdout:(32/33): ceph-mgr-17.2.7-521.gc80e7168.el9.x86_ 447 kB/s | 1.4 MB 00:03 2024-02-20T07:48:33.279 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:33.279 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:33.280 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:33.280 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:33.280 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:33.280 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:33.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T07:48:34.037 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:06 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:34.210 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:34.211 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:34.211 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-02-20T07:48:34.211 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:34.211 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-521.gc80e7168.el9 ceph 137 k 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 137 k 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 379 k 2024-02-20T07:48:34.212 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:34.442 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-17.2.7-521.gc80e716 596 kB/s | 137 kB 00:00 2024-02-20T07:48:34.442 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:34.443 INFO:teuthology.orchestra.run.smithi055.stdout:Total 593 kB/s | 137 kB 00:00 2024-02-20T07:48:34.443 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:34.451 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:34.451 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:34.504 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:34.504 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:34.652 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:34.719 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:48:34.743 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:48:34.743 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:34.743 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-20T07:48:34.743 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:35.189 INFO:teuthology.orchestra.run.smithi186.stdout:(33/33): ceph-mon-17.2.7-521.gc80e7168.el9.x86_ 783 kB/s | 3.7 MB 00:04 2024-02-20T07:48:35.192 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:35.192 INFO:teuthology.orchestra.run.smithi186.stdout:Total 4.9 MB/s | 27 MB 00:05 2024-02-20T07:48:35.193 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:48:35.232 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:48:35.232 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:48:35.585 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:48:35.586 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:48:35.718 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:48:35.718 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:35.718 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:35.718 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:35.718 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:35.718 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:35.930 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2024-02-20T07:48:36.518 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:08 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:36.638 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:48:36.691 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:36.692 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:48:36.692 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:36.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T07:48:36.801 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:48:36.883 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-20T07:48:36.916 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-20T07:48:36.948 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-20T07:48:37.040 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-20T07:48:37.110 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-20T07:48:37.152 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-20T07:48:37.223 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-20T07:48:37.277 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-20T07:48:37.315 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-20T07:48:37.336 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:09 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:37.364 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-20T07:48:37.428 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-20T07:48:37.508 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:37.509 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:37.509 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:37.509 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:37.509 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 1.7 M 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 23 k 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 15 k 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout:Install 7 Packages 2024-02-20T07:48:37.510 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:37.511 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2024-02-20T07:48:37.511 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 8.1 M 2024-02-20T07:48:37.511 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:37.612 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-20T07:48:37.661 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-20T07:48:37.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-20T07:48:37.727 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-20T07:48:37.756 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-20T07:48:37.824 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-20T07:48:37.858 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:48:37.892 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-20T07:48:37.932 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-20T07:48:37.939 INFO:teuthology.orchestra.run.smithi055.stdout:(1/7): ceph-prometheus-alerts-17.2.7-521.gc80e7 76 kB/s | 15 kB 00:00 2024-02-20T07:48:37.964 INFO:teuthology.orchestra.run.smithi055.stdout:(2/7): ceph-grafana-dashboards-17.2.7-521.gc80e 107 kB/s | 23 kB 00:00 2024-02-20T07:48:37.967 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-20T07:48:38.002 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-20T07:48:38.014 INFO:teuthology.orchestra.run.smithi055.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2024-02-20T07:48:38.040 INFO:teuthology.orchestra.run.smithi055.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 542 kB/s | 41 kB 00:00 2024-02-20T07:48:38.066 INFO:teuthology.orchestra.run.smithi055.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 607 kB/s | 31 kB 00:00 2024-02-20T07:48:38.090 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-20T07:48:38.100 INFO:teuthology.orchestra.run.smithi055.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-02-20T07:48:38.128 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-20T07:48:38.196 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-20T07:48:38.342 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-20T07:48:38.464 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:48:38.532 INFO:teuthology.orchestra.run.smithi055.stdout:(7/7): ceph-mgr-dashboard-17.2.7-521.gc80e7168. 2.2 MB/s | 1.7 MB 00:00 2024-02-20T07:48:38.534 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:38.534 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-02-20T07:48:38.535 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:38.544 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:38.544 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:38.615 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 28/33 2024-02-20T07:48:38.617 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:38.617 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:38.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:48:38.671 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:48:38.671 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:38.672 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-20T07:48:38.672 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:48:38.672 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:48:38.672 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:38.924 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:38.992 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-20T07:48:39.027 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-20T07:48:39.082 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-20T07:48:39.121 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-20T07:48:39.150 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 5/7 2024-02-20T07:48:39.391 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 6/7 2024-02-20T07:48:39.419 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:48:39.950 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:48:39.950 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 1/7 2024-02-20T07:48:39.950 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2/7 2024-02-20T07:48:39.950 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 3/7 2024-02-20T07:48:39.950 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-20T07:48:39.951 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-20T07:48:39.951 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-20T07:48:40.073 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:48:40.101 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:48:40.101 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:40.101 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-20T07:48:40.101 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:48:40.101 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:48:40.101 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:40.282 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-20T07:48:40.283 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:40.283 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:40.283 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:40.283 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:40.283 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:40.284 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-20T07:48:40.284 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-20T07:48:40.284 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-20T07:48:40.284 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-20T07:48:40.284 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:40.284 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:40.490 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:48:40.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T07:48:40.514 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:48:40.514 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:40.514 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-20T07:48:40.514 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:48:40.514 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:48:40.514 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:40.716 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:48:40.739 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:48:40.739 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:40.739 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-20T07:48:40.739 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:48:40.739 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:48:40.739 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:40.804 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:48:41.086 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:13 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:41.261 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:41.262 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================== 2024-02-20T07:48:41.262 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:41.262 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================== 2024-02-20T07:48:41.262 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:41.262 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 7.4 M 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-20T07:48:41.263 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================== 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:41.264 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2024-02-20T07:48:41.265 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 209 M 2024-02-20T07:48:41.265 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:41.439 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 241 kB/s | 32 kB 00:00 2024-02-20T07:48:41.473 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 542 kB/s | 17 kB 00:00 2024-02-20T07:48:41.548 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 377 kB/s | 34 kB 00:00 2024-02-20T07:48:41.732 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-02-20T07:48:41.816 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-02-20T07:48:41.841 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-02-20T07:48:42.000 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 802 kB 00:00 2024-02-20T07:48:42.175 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-02-20T07:48:42.200 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 982 kB/s | 194 kB 00:00 2024-02-20T07:48:42.367 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-02-20T07:48:42.567 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 1/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 3/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 4/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 5/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 6/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-20T07:48:42.568 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-20T07:48:42.569 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:48:42.570 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-20T07:48:42.570 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-20T07:48:42.570 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-20T07:48:42.570 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-20T07:48:42.570 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-20T07:48:42.701 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 5.3 MB/s | 7.4 MB 00:01 2024-02-20T07:48:42.703 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:42.704 INFO:teuthology.orchestra.run.smithi055.stdout:Total 26 MB/s | 37 MB 00:01 2024-02-20T07:48:42.722 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:42.749 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:42.750 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:43.197 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:43.197 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:43.392 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-20T07:48:43.392 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:43.392 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-20T07:48:43.393 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-20T07:48:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:43.395 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:43.728 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base 2024-02-20T07:48:44.039 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:44.109 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-20T07:48:44.138 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-20T07:48:44.237 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:48:44.275 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-20T07:48:44.315 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:08 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:44.427 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:48:44.459 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:48:44.477 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:44.478 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:48:44.478 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:44.490 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:48:44.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install cephadm 2024-02-20T07:48:44.579 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-20T07:48:45.110 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:09 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:45.273 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-20T07:48:45.275 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:45.275 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 80 k 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:48:45.276 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:45.277 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 80 k 2024-02-20T07:48:45.277 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 364 k 2024-02-20T07:48:45.277 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:48:45.390 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-20T07:48:45.490 INFO:teuthology.orchestra.run.smithi186.stdout:cephadm-17.2.7-521.gc80e7168.el9.noarch.rpm 379 kB/s | 80 kB 00:00 2024-02-20T07:48:45.491 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:45.491 INFO:teuthology.orchestra.run.smithi186.stdout:Total 376 kB/s | 80 kB 00:00 2024-02-20T07:48:45.491 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:48:45.496 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:48:45.496 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:48:45.497 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:48:45.498 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:48:45.572 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:48:46.042 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:46.145 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:46.505 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:46.691 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-20T07:48:46.771 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:48:46.771 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:46.771 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:48:46.771 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:46.771 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:46.771 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:46.995 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T07:48:47.588 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:11 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:47.756 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:47.757 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:47.757 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-02-20T07:48:47.757 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:47.757 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-521.gc80e7168.el9 ceph 137 k 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 137 k 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 379 k 2024-02-20T07:48:47.758 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:48:47.783 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-20T07:48:47.815 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:48:48.119 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-17.2.7-521.gc80e716 380 kB/s | 137 kB 00:00 2024-02-20T07:48:48.119 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:48.120 INFO:teuthology.orchestra.run.smithi186.stdout:Total 378 kB/s | 137 kB 00:00 2024-02-20T07:48:48.120 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:48:48.127 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:48:48.127 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:48:48.179 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:48:48.180 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:48:48.323 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:48:48.376 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:48:48.400 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:48:48.401 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:48:48.401 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-20T07:48:48.401 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:49.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:48:49.420 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:49.420 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:48:49.420 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:48:49.420 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:49.420 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:49.673 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr 2024-02-20T07:48:50.266 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:14 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:50.385 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:48:50.436 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:50.437 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:48:50.437 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:50.469 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:48:50.469 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 1/12 2024-02-20T07:48:50.469 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-20T07:48:50.469 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:48:50.469 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-20T07:48:50.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-20T07:48:50.495 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-20T07:48:51.018 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:51.019 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:51.067 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:15 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:51.233 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 1.7 M 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 23 k 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 15 k 2024-02-20T07:48:51.234 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout:Install 7 Packages 2024-02-20T07:48:51.235 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:51.236 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.0 M 2024-02-20T07:48:51.236 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 8.1 M 2024-02-20T07:48:51.236 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:48:51.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2024-02-20T07:48:51.779 INFO:teuthology.orchestra.run.smithi186.stdout:(1/7): ceph-prometheus-alerts-17.2.7-521.gc80e7 79 kB/s | 15 kB 00:00 2024-02-20T07:48:51.921 INFO:teuthology.orchestra.run.smithi186.stdout:(2/7): ceph-grafana-dashboards-17.2.7-521.gc80e 71 kB/s | 23 kB 00:00 2024-02-20T07:48:52.007 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:24 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:52.181 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:52.182 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-02-20T07:48:52.182 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:52.182 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 47 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-20T07:48:52.183 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:52.184 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2024-02-20T07:48:52.185 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2024-02-20T07:48:52.185 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:52.221 INFO:teuthology.orchestra.run.smithi186.stdout:(3/7): ceph-mgr-dashboard-17.2.7-521.gc80e7168. 2.7 MB/s | 1.7 MB 00:00 2024-02-20T07:48:52.474 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 24 kB 00:00 2024-02-20T07:48:52.488 INFO:teuthology.orchestra.run.smithi186.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 13 kB/s | 9.0 kB 00:00 2024-02-20T07:48:52.499 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-02-20T07:48:52.550 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-20T07:48:52.567 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-02-20T07:48:52.592 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-02-20T07:48:52.617 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-02-20T07:48:52.651 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): ceph-mgr-rook-17.2.7-521.gc80e7168.el9. 153 kB/s | 47 kB 00:00 2024-02-20T07:48:52.851 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-02-20T07:48:52.930 INFO:teuthology.orchestra.run.smithi186.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 41 kB/s | 41 kB 00:01 2024-02-20T07:48:53.001 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 79 kB/s | 32 kB 00:00 2024-02-20T07:48:53.085 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-02-20T07:48:53.135 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-02-20T07:48:53.160 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-02-20T07:48:53.230 INFO:teuthology.orchestra.run.smithi186.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 30 kB/s | 31 kB 00:01 2024-02-20T07:48:53.402 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-google-auth-2.27.0-1.el9.noarc 279 kB/s | 203 kB 00:00 2024-02-20T07:48:53.407 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:53.407 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-02-20T07:48:53.407 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:53.422 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:53.422 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:53.430 INFO:teuthology.orchestra.run.smithi186.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 200 kB/s | 188 kB 00:00 2024-02-20T07:48:53.435 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:53.435 INFO:teuthology.orchestra.run.smithi186.stdout:Total 929 kB/s | 2.0 MB 00:02 2024-02-20T07:48:53.436 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:48:53.444 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:48:53.445 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:48:53.518 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:48:53.519 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:48:53.529 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:53.531 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:53.823 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:48:53.890 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-20T07:48:53.923 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-20T07:48:53.970 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-20T07:48:54.003 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-20T07:48:54.031 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 5/7 2024-02-20T07:48:54.056 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:54.232 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-20T07:48:54.261 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 6/7 2024-02-20T07:48:54.297 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-20T07:48:54.301 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:48:54.353 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-20T07:48:54.396 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-20T07:48:54.447 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-20T07:48:54.545 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-20T07:48:54.654 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-20T07:48:54.707 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-20T07:48:54.857 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:48:54.858 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 1/7 2024-02-20T07:48:54.858 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2/7 2024-02-20T07:48:54.858 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 3/7 2024-02-20T07:48:54.858 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-20T07:48:54.858 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-20T07:48:54.858 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-20T07:48:55.188 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-20T07:48:55.188 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:55.188 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:48:55.188 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:55.188 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:55.188 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:55.189 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-20T07:48:55.189 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-20T07:48:55.189 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-20T07:48:55.189 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-20T07:48:55.189 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:55.189 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:48:55.422 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T07:48:55.468 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-20T07:48:55.536 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-20T07:48:55.570 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-20T07:48:55.617 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-20T07:48:55.640 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:48:55.995 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:19 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:48:56.163 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:48:56.163 INFO:teuthology.orchestra.run.smithi186.stdout:=========================================================================================== 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout:=========================================================================================== 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 7.4 M 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-20T07:48:56.164 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout:=========================================================================================== 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-02-20T07:48:56.165 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:48:56.166 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 37 M 2024-02-20T07:48:56.166 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 209 M 2024-02-20T07:48:56.166 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 1/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-20T07:48:56.470 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-20T07:48:56.471 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-20T07:48:56.471 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-20T07:48:56.471 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-20T07:48:56.471 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-20T07:48:56.471 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-20T07:48:56.579 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2024-02-20T07:48:56.604 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 726 kB/s | 17 kB 00:00 2024-02-20T07:48:56.630 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-02-20T07:48:56.697 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-20T07:48:56.872 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-02-20T07:48:56.939 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-20T07:48:56.964 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-20T07:48:56.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-20T07:48:56.972 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:56.972 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:48:56.972 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:48:56.972 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:56.973 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:48:57.148 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 802 kB 00:00 2024-02-20T07:48:57.332 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-02-20T07:48:57.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T07:48:57.356 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 943 kB/s | 194 kB 00:00 2024-02-20T07:48:57.448 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-02-20T07:48:57.749 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 5.7 MB/s | 7.4 MB 00:01 2024-02-20T07:48:57.752 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:57.752 INFO:teuthology.orchestra.run.smithi186.stdout:Total 23 MB/s | 37 MB 00:01 2024-02-20T07:48:57.768 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:48:57.794 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:48:57.795 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:48:57.919 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:29 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:48:58.093 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 121 k 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-20T07:48:58.094 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-02-20T07:48:58.095 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:48:58.096 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.8 M 2024-02-20T07:48:58.096 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 33 M 2024-02-20T07:48:58.096 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:48:58.233 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:48:58.233 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:48:58.329 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-02-20T07:48:58.480 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-02-20T07:48:58.505 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 3.1 MB/s | 548 kB 00:00 2024-02-20T07:48:58.538 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-02-20T07:48:58.564 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 690 kB/s | 58 kB 00:00 2024-02-20T07:48:58.589 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-mgr-cephadm-17.2.7-521.gc80e7168.el 284 kB/s | 121 kB 00:00 2024-02-20T07:48:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:58.591 INFO:teuthology.orchestra.run.smithi055.stdout:Total 14 MB/s | 6.8 MB 00:00 2024-02-20T07:48:58.591 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:48:58.600 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:48:58.600 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:48:58.693 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:48:58.693 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:48:58.942 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:48:58.998 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:48:59.009 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-20T07:48:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-20T07:48:59.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:48:59.173 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-20T07:48:59.187 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-20T07:48:59.258 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-20T07:48:59.354 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:48:59.381 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:48:59.464 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-20T07:48:59.807 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-20T07:48:59.886 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-20T07:48:59.941 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-20T07:48:59.978 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:49:00.136 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-20T07:49:00.243 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-20T07:49:00.801 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:49:00.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/6 2024-02-20T07:49:00.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-20T07:49:00.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-20T07:49:00.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-20T07:49:00.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-20T07:49:01.130 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-20T07:49:01.130 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:01.130 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:01.131 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:49:01.131 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-20T07:49:01.131 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-20T07:49:01.131 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-20T07:49:01.132 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-20T07:49:01.132 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-20T07:49:01.132 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:01.133 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:01.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2024-02-20T07:49:01.514 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-20T07:49:01.956 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:33 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:02.131 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 812 k 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-20T07:49:02.132 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:02.133 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:49:02.133 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:02.133 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-02-20T07:49:02.133 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:02.134 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 891 k 2024-02-20T07:49:02.134 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.6 M 2024-02-20T07:49:02.134 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:49:02.283 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 528 kB/s | 79 kB 00:00 2024-02-20T07:49:02.545 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-20T07:49:02.573 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:49:02.674 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-fuse-17.2.7-521.gc80e7168.el9.x86_6 1.5 MB/s | 812 kB 00:00 2024-02-20T07:49:02.676 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:02.676 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.6 MB/s | 891 kB 00:00 2024-02-20T07:49:02.676 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:49:02.699 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:49:02.699 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:49:02.777 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:49:02.777 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:49:02.963 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:49:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-20T07:49:03.160 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:49:03.842 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:49:03.842 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/2 2024-02-20T07:49:04.189 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-20T07:49:04.189 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:04.189 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:04.189 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-20T07:49:04.189 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:04.189 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:04.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2024-02-20T07:49:05.054 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:37 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:05.153 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:49:05.153 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 1/12 2024-02-20T07:49:05.153 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-20T07:49:05.153 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:49:05.154 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-20T07:49:05.154 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:49:05.154 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:49:05.154 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-20T07:49:05.154 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-20T07:49:05.154 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-20T07:49:05.155 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-20T07:49:05.155 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-20T07:49:05.227 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 126 k 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:05.228 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 126 k 2024-02-20T07:49:05.229 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 448 k 2024-02-20T07:49:05.229 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:49:05.454 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-17.2.7-521.gc80e7168.el9.x86_64. 562 kB/s | 126 kB 00:00 2024-02-20T07:49:05.454 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:05.454 INFO:teuthology.orchestra.run.smithi055.stdout:Total 558 kB/s | 126 kB 00:00 2024-02-20T07:49:05.455 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:49:05.460 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:49:05.460 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:49:05.518 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:49:05.518 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:49:05.642 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-20T07:49:05.642 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:05.642 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:05.642 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:49:05.642 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:05.643 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:05.667 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:49:05.779 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:06.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-rook 2024-02-20T07:49:06.226 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:06.486 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:06.486 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:06.486 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:06.486 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:06.486 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:06.486 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:06.590 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:30 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:06.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2024-02-20T07:49:06.758 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:06.759 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-02-20T07:49:06.759 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:49:06.759 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-02-20T07:49:06.759 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 47 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-20T07:49:06.760 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-20T07:49:06.761 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-20T07:49:06.761 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:06.761 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:06.761 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-02-20T07:49:06.761 INFO:teuthology.orchestra.run.smithi186.stdout:Install 13 Packages 2024-02-20T07:49:06.761 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:06.762 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.0 M 2024-02-20T07:49:06.762 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-02-20T07:49:06.762 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:07.322 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:39 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:07.339 INFO:teuthology.orchestra.run.smithi186.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-02-20T07:49:07.365 INFO:teuthology.orchestra.run.smithi186.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 97 kB/s | 18 kB 00:00 2024-02-20T07:49:07.390 INFO:teuthology.orchestra.run.smithi186.stdout:(3/13): ceph-mgr-rook-17.2.7-521.gc80e7168.el9. 224 kB/s | 47 kB 00:00 2024-02-20T07:49:07.441 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:07.482 INFO:teuthology.orchestra.run.smithi186.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-02-20T07:49:07.496 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:07.497 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:07.497 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:07.507 INFO:teuthology.orchestra.run.smithi186.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 929 kB/s | 132 kB 00:00 2024-02-20T07:49:07.541 INFO:teuthology.orchestra.run.smithi186.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 211 kB 00:00 2024-02-20T07:49:07.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2024-02-20T07:49:07.591 INFO:teuthology.orchestra.run.smithi186.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 394 kB/s | 43 kB 00:00 2024-02-20T07:49:07.783 INFO:teuthology.orchestra.run.smithi186.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 68 kB/s | 15 kB 00:00 2024-02-20T07:49:07.808 INFO:teuthology.orchestra.run.smithi186.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-02-20T07:49:08.140 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:40 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:08.312 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:08.313 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:08.313 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-02-20T07:49:08.313 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:08.313 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:49:08.313 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:49:08.313 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 25 k 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 110 k 2024-02-20T07:49:08.314 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:49:08.483 INFO:teuthology.orchestra.run.smithi186.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 227 kB/s | 203 kB 00:00 2024-02-20T07:49:08.684 INFO:teuthology.orchestra.run.smithi186.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 67 kB/s | 59 kB 00:00 2024-02-20T07:49:08.917 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-17.2.7-521.gc80e7168.el9.x86_64 41 kB/s | 25 kB 00:00 2024-02-20T07:49:08.918 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:08.918 INFO:teuthology.orchestra.run.smithi055.stdout:Total 41 kB/s | 25 kB 00:00 2024-02-20T07:49:08.918 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:49:08.922 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:49:08.922 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:49:08.924 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:49:08.924 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:49:08.950 INFO:teuthology.orchestra.run.smithi186.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 192 kB/s | 90 kB 00:00 2024-02-20T07:49:09.002 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:49:09.102 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:09.494 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:09.609 INFO:teuthology.orchestra.run.smithi186.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 572 kB/s | 1.0 MB 00:01 2024-02-20T07:49:09.615 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:09.615 INFO:teuthology.orchestra.run.smithi186.stdout:Total 733 kB/s | 2.0 MB 00:02 2024-02-20T07:49:09.616 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:09.631 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:09.631 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:09.739 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:09.739 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:09.769 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:09.769 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:09.769 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:09.769 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:09.769 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:09.769 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:09.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2024-02-20T07:49:10.322 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:10.491 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-20T07:49:10.542 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:42 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:10.543 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-20T07:49:10.599 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-20T07:49:10.637 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-20T07:49:10.660 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:10.678 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-20T07:49:10.714 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:10.715 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:10.715 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:10.772 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-20T07:49:10.781 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2024-02-20T07:49:10.872 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-20T07:49:10.929 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-20T07:49:11.368 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:43 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:11.487 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:11.543 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:11.544 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:11.544 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:11.603 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2024-02-20T07:49:11.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-20T07:49:11.710 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-20T07:49:11.744 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-20T07:49:11.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-20T07:49:11.814 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:49:12.186 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:44 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:12.308 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:12.362 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:12.363 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:12.363 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:12.423 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 1/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-20T07:49:12.673 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-20T07:49:12.674 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-20T07:49:12.674 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-20T07:49:13.006 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:45 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:13.063 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-20T07:49:13.063 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:13.063 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:13.063 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:49:13.064 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-20T07:49:13.064 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-20T07:49:13.064 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-20T07:49:13.064 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-20T07:49:13.064 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-20T07:49:13.065 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-20T07:49:13.065 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-20T07:49:13.065 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-20T07:49:13.065 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-20T07:49:13.065 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-20T07:49:13.065 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-20T07:49:13.066 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-20T07:49:13.066 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:13.066 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:13.124 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:13.180 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:13.180 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:13.180 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:13.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2024-02-20T07:49:13.284 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T07:49:13.833 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:45 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:13.863 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:37 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:13.952 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:14.007 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:14.009 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:14.009 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:14.029 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:14.029 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-02-20T07:49:14.029 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:49:14.029 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-02-20T07:49:14.029 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:14.029 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 121 k 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-02-20T07:49:14.030 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:14.031 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.8 M 2024-02-20T07:49:14.031 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 33 M 2024-02-20T07:49:14.031 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:14.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2024-02-20T07:49:14.387 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 227 kB 00:00 2024-02-20T07:49:14.620 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-02-20T07:49:14.646 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): ceph-mgr-cephadm-17.2.7-521.gc80e7168.el 267 kB/s | 121 kB 00:00 2024-02-20T07:49:14.659 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:46 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:14.777 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:14.832 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:14.833 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-02-20T07:49:14.833 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:14.837 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 267 kB/s | 58 kB 00:00 2024-02-20T07:49:14.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2024-02-20T07:49:14.930 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-02-20T07:49:14.963 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-02-20T07:49:14.966 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:14.966 INFO:teuthology.orchestra.run.smithi186.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-02-20T07:49:14.967 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:15.065 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:15.065 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:15.364 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:15.482 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:47 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:15.535 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-20T07:49:15.597 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-20T07:49:15.656 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:15.657 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:15.657 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-02-20T07:49:15.657 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:15.657 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:49:15.657 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 81 k 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 81 k 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 222 k 2024-02-20T07:49:15.658 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:49:15.879 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-17.2.7-521.gc80e7168.el9.x86_64.rpm 368 kB/s | 81 kB 00:00 2024-02-20T07:49:15.879 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:15.880 INFO:teuthology.orchestra.run.smithi055.stdout:Total 366 kB/s | 81 kB 00:00 2024-02-20T07:49:15.880 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:49:15.886 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:49:15.887 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:49:15.944 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:49:15.944 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:49:16.086 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:49:16.108 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-20T07:49:16.188 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:16.190 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-20T07:49:16.255 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-20T07:49:16.286 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:49:16.708 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:17.028 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:17.028 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:17.028 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:17.028 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:17.028 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:17.029 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:17.037 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:49:17.037 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/6 2024-02-20T07:49:17.037 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-20T07:49:17.037 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-20T07:49:17.037 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-20T07:49:17.037 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-20T07:49:17.272 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2024-02-20T07:49:17.368 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-20T07:49:17.368 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:17.368 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:17.369 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:17.611 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse 2024-02-20T07:49:17.859 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:49 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:18.034 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:18.034 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:18.035 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:49:18.036 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:18.036 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.9 M 2024-02-20T07:49:18.036 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 12 M 2024-02-20T07:49:18.036 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:49:18.199 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:42 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:18.367 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:18.368 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:18.368 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-02-20T07:49:18.368 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 812 k 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 891 k 2024-02-20T07:49:18.369 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.6 M 2024-02-20T07:49:18.370 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:18.490 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 653 kB/s | 79 kB 00:00 2024-02-20T07:49:18.558 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-17.2.7-521.gc80e7168.el9.x86_64.rpm 5.6 MB/s | 2.9 MB 00:00 2024-02-20T07:49:18.558 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:18.558 INFO:teuthology.orchestra.run.smithi055.stdout:Total 5.6 MB/s | 2.9 MB 00:00 2024-02-20T07:49:18.558 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:49:18.568 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:49:18.568 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:49:18.643 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:49:18.644 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:49:18.707 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-fuse-17.2.7-521.gc80e7168.el9.x86_6 2.3 MB/s | 812 kB 00:00 2024-02-20T07:49:18.708 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:18.708 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.6 MB/s | 891 kB 00:00 2024-02-20T07:49:18.709 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:18.729 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:18.729 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:18.805 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:18.805 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:18.989 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:19.076 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:49:19.127 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-20T07:49:19.133 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:19.157 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:19.157 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:49:19.157 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-20T07:49:19.157 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:49:19.157 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:49:19.157 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:19.190 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:49:19.923 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:49:19.923 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/2 2024-02-20T07:49:20.251 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-20T07:49:20.251 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:20.251 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:20.251 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-20T07:49:20.251 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:20.251 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:20.285 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:20.285 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:20.285 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:20.285 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:20.285 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:20.285 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:20.506 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel 2024-02-20T07:49:20.517 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2024-02-20T07:49:21.099 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:45 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:21.118 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:53 ago on Tue 20 Feb 2024 07:47:28 AM UTC. 2024-02-20T07:49:21.267 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:21.268 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:21.268 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-02-20T07:49:21.268 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:21.268 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:21.268 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 126 k 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 126 k 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 448 k 2024-02-20T07:49:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:21.292 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-02-20T07:49:21.292 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 165 k 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 165 k 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 477 k 2024-02-20T07:49:21.293 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-02-20T07:49:21.662 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-17.2.7-521.gc80e7168.el9.x86_64. 322 kB/s | 126 kB 00:00 2024-02-20T07:49:21.662 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:21.662 INFO:teuthology.orchestra.run.smithi186.stdout:Total 321 kB/s | 126 kB 00:00 2024-02-20T07:49:21.663 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:21.668 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:21.668 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:21.722 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:21.723 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:21.760 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-17.2.7-521.gc80e7168.el9.x86_64.rpm 353 kB/s | 165 kB 00:00 2024-02-20T07:49:21.761 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:21.761 INFO:teuthology.orchestra.run.smithi055.stdout:Total 352 kB/s | 165 kB 00:00 2024-02-20T07:49:21.761 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-02-20T07:49:21.770 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-02-20T07:49:21.770 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-02-20T07:49:21.828 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-02-20T07:49:21.828 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-02-20T07:49:21.871 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:21.959 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:21.976 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-02-20T07:49:22.086 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:22.421 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:22.496 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:22.726 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:22.726 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:22.726 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:22.726 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:22.726 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:22.727 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:22.808 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:22.808 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:22.808 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-02-20T07:49:22.808 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:22.808 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:49:22.808 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-02-20T07:49:22.982 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2 2024-02-20T07:49:23.063 DEBUG:teuthology.parallel:result is None 2024-02-20T07:49:23.576 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:47 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:23.697 INFO:teuthology.orchestra.run.smithi186.stdout:Package libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:23.748 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:23.749 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:23.749 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:23.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs-devel 2024-02-20T07:49:24.399 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:48 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:24.564 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:24.565 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:24.566 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:24.566 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:49:24.566 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:24.566 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 25 k 2024-02-20T07:49:24.566 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 110 k 2024-02-20T07:49:24.566 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:24.725 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs-devel-17.2.7-521.gc80e7168.el9.x86_64 157 kB/s | 25 kB 00:00 2024-02-20T07:49:24.726 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:24.726 INFO:teuthology.orchestra.run.smithi186.stdout:Total 156 kB/s | 25 kB 00:00 2024-02-20T07:49:24.726 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:24.729 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:24.729 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:24.731 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:24.731 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:24.820 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:24.927 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:25.319 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:25.599 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:25.599 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:25.599 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:25.599 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:25.599 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:25.599 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:25.766 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2 2024-02-20T07:49:26.345 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:50 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:26.460 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:26.512 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:26.513 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:26.513 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:26.573 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1 2024-02-20T07:49:27.150 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:51 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:27.264 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:27.316 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:27.317 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:27.317 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:27.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados 2024-02-20T07:49:27.960 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:51 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:28.073 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:28.125 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:28.126 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:28.126 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:28.190 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw 2024-02-20T07:49:28.763 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:52 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:28.877 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:28.929 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:28.930 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:28.930 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:28.990 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs 2024-02-20T07:49:29.569 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:53 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:29.683 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:29.736 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:29.736 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:29.737 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:29.800 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd 2024-02-20T07:49:30.384 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:30.498 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:49:30.550 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:30.551 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-02-20T07:49:30.551 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:30.618 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse 2024-02-20T07:49:31.195 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:55 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:31.360 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:31.360 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-02-20T07:49:31.360 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 81 k 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:31.361 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 81 k 2024-02-20T07:49:31.362 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 222 k 2024-02-20T07:49:31.362 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:31.569 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-17.2.7-521.gc80e7168.el9.x86_64.rpm 391 kB/s | 81 kB 00:00 2024-02-20T07:49:31.570 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:31.570 INFO:teuthology.orchestra.run.smithi186.stdout:Total 389 kB/s | 81 kB 00:00 2024-02-20T07:49:31.570 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:31.577 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:31.578 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:31.631 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:31.631 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:31.765 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:31.875 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:32.303 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:32.597 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:32.598 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:32.598 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:32.598 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:32.598 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:32.598 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:32.795 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror 2024-02-20T07:49:33.382 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:57 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:33.554 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:33.555 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:33.555 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-02-20T07:49:33.555 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:33.555 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:33.555 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:49:33.555 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.9 M 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 12 M 2024-02-20T07:49:33.556 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:34.308 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-17.2.7-521.gc80e7168.el9.x86_64.rpm 3.9 MB/s | 2.9 MB 00:00 2024-02-20T07:49:34.308 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:34.309 INFO:teuthology.orchestra.run.smithi186.stdout:Total 3.9 MB/s | 2.9 MB 00:00 2024-02-20T07:49:34.309 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:34.318 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:34.318 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:34.389 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:34.389 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:34.800 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:34.859 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:34.883 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:34.883 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:49:34.883 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-20T07:49:34.883 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:49:34.883 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:49:34.883 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:36.025 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:36.025 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:36.025 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:36.025 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:36.025 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:36.025 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:36.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd 2024-02-20T07:49:36.859 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:00 ago on Tue 20 Feb 2024 07:47:36 AM UTC. 2024-02-20T07:49:37.025 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 165 k 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 165 k 2024-02-20T07:49:37.026 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 477 k 2024-02-20T07:49:37.027 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-02-20T07:49:37.378 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-17.2.7-521.gc80e7168.el9.x86_64.rpm 469 kB/s | 165 kB 00:00 2024-02-20T07:49:37.379 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:49:37.379 INFO:teuthology.orchestra.run.smithi186.stdout:Total 467 kB/s | 165 kB 00:00 2024-02-20T07:49:37.379 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-02-20T07:49:37.387 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-02-20T07:49:37.387 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-02-20T07:49:37.442 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-02-20T07:49:37.443 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-02-20T07:49:37.591 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-02-20T07:49:37.701 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:38.132 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:38.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:49:38.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:38.431 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-02-20T07:49:38.431 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:49:38.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:49:38.431 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-02-20T07:49:38.666 DEBUG:teuthology.parallel:result is None 2024-02-20T07:49:38.667 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:49:38.667 INFO:teuthology.packaging:ref: None 2024-02-20T07:49:38.667 INFO:teuthology.packaging:tag: None 2024-02-20T07:49:38.667 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:49:38.667 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:49:38.667 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:49:38.905 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T07:49:38.929 INFO:teuthology.orchestra.run.smithi055.stdout:17.2.7-521.gc80e7168.el9 2024-02-20T07:49:38.930 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-521.gc80e7168.el9 2024-02-20T07:49:38.930 INFO:teuthology.task.install:The correct ceph version 17.2.7-521.gc80e7168 is installed. 2024-02-20T07:49:38.931 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:49:38.932 INFO:teuthology.packaging:ref: None 2024-02-20T07:49:38.932 INFO:teuthology.packaging:tag: None 2024-02-20T07:49:38.932 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:49:38.932 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:49:38.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:49:39.256 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T07:49:39.277 INFO:teuthology.orchestra.run.smithi186.stdout:17.2.7-521.gc80e7168.el9 2024-02-20T07:49:39.277 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-521.gc80e7168.el9 2024-02-20T07:49:39.277 INFO:teuthology.task.install:The correct ceph version 17.2.7-521.gc80e7168 is installed. 2024-02-20T07:49:39.279 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-20T07:49:39.279 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:49:39.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T07:49:39.311 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:49:39.311 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T07:49:39.349 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-20T07:49:39.349 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:49:39.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T07:49:39.381 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T07:49:39.450 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:49:39.451 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T07:49:39.478 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T07:49:39.546 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-20T07:49:39.546 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:49:39.546 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T07:49:39.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T07:49:39.644 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:49:39.644 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T07:49:39.671 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T07:49:39.738 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-20T07:49:39.738 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:49:39.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T07:49:39.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T07:49:39.834 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:49:39.835 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T07:49:39.861 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T07:49:39.928 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:49:39.938 INFO:teuthology.task.print:**** done install task... 2024-02-20T07:49:39.938 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-20T07:49:40.042 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c'} 2024-02-20T07:49:40.042 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-20T07:49:40.043 INFO:tasks.cephadm:Cluster fsid is 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:49:40.043 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-20T07:49:40.043 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-20T07:49:40.043 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi055': '172.21.15.55', 'mon.smithi186': '172.21.15.186'} 2024-02-20T07:49:40.044 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-20T07:49:40.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2024-02-20T07:49:40.073 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-02-20T07:49:40.100 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-20T07:49:40.100 DEBUG:teuthology.orchestra.run.smithi055:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T07:49:40.278 INFO:teuthology.orchestra.run.smithi055.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 20 07:49 /home/ubuntu/cephtest/cephadm 2024-02-20T07:49:40.278 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T07:49:40.464 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 20 07:49 /home/ubuntu/cephtest/cephadm 2024-02-20T07:49:40.464 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T07:49:40.482 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T07:49:40.513 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-20T07:49:40.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-20T07:49:40.525 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-20T07:49:41.369 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:49:41.448 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:50:08.532 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-02-20T07:50:08.533 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)", 2024-02-20T07:50:08.533 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e", 2024-02-20T07:50:08.533 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-02-20T07:50:08.533 INFO:teuthology.orchestra.run.smithi186.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b" 2024-02-20T07:50:08.533 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-02-20T07:50:08.533 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-02-20T07:50:14.450 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-02-20T07:50:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)", 2024-02-20T07:50:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "image_id": "10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e", 2024-02-20T07:50:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "repo_digests": [ 2024-02-20T07:50:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b" 2024-02-20T07:50:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-02-20T07:50:14.451 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-02-20T07:50:14.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2024-02-20T07:50:14.510 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-02-20T07:50:14.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2024-02-20T07:50:14.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-02-20T07:50:14.616 INFO:tasks.cephadm:Writing seed config... 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-20T07:50:14.617 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-20T07:50:14.618 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-20T07:50:14.619 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-20T07:50:14.620 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-20T07:50:14.620 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-20T07:50:14.620 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-20T07:50:14.620 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:50:14.620 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-20T07:50:14.635 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9a1162a4-cfc4-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-20T07:50:14.636 DEBUG:teuthology.orchestra.run.smithi055:mon.smithi055> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service 2024-02-20T07:50:14.678 INFO:tasks.cephadm:Bootstrapping... 2024-02-20T07:50:14.678 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.55 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-20T07:50:14.887 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:50:14.888 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '9a1162a4-cfc4-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.55', '--skip-admin-label'] 2024-02-20T07:50:14.912 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-20T07:50:14.912 INFO:teuthology.orchestra.run.smithi055.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-20T07:50:14.913 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying podman|docker is present... 2024-02-20T07:50:14.938 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-20T07:50:14.938 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying lvm2 is present... 2024-02-20T07:50:14.938 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying time synchronization is in place... 2024-02-20T07:50:14.946 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-20T07:50:14.946 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T07:50:14.952 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-20T07:50:14.952 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-02-20T07:50:14.959 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout enabled 2024-02-20T07:50:14.965 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout active 2024-02-20T07:50:14.965 INFO:teuthology.orchestra.run.smithi055.stdout:Unit chronyd.service is enabled and running 2024-02-20T07:50:14.965 INFO:teuthology.orchestra.run.smithi055.stdout:Repeating the final host check... 2024-02-20T07:50:14.988 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-20T07:50:14.988 INFO:teuthology.orchestra.run.smithi055.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-20T07:50:14.988 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl is present 2024-02-20T07:50:14.988 INFO:teuthology.orchestra.run.smithi055.stdout:lvcreate is present 2024-02-20T07:50:14.994 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-20T07:50:14.994 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T07:50:15.000 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-20T07:50:15.000 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-02-20T07:50:15.007 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout enabled 2024-02-20T07:50:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout active 2024-02-20T07:50:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:Unit chronyd.service is enabled and running 2024-02-20T07:50:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:Host looks OK 2024-02-20T07:50:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:Cluster fsid: 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:15.014 INFO:teuthology.orchestra.run.smithi055.stdout:Acquiring lock 139713496660480 on /run/cephadm/9a1162a4-cfc4-11ee-95bb-87774f69a715.lock 2024-02-20T07:50:15.014 INFO:teuthology.orchestra.run.smithi055.stdout:Lock 139713496660480 acquired on /run/cephadm/9a1162a4-cfc4-11ee-95bb-87774f69a715.lock 2024-02-20T07:50:15.014 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying IP 172.21.15.55 port 3300 ... 2024-02-20T07:50:15.014 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying IP 172.21.15.55 port 6789 ... 2024-02-20T07:50:15.014 INFO:teuthology.orchestra.run.smithi055.stdout:Base mon IP(s) is [172.21.15.55:3300, 172.21.15.55:6789], mon addrv is [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2024-02-20T07:50:15.018 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.55 metric 100 2024-02-20T07:50:15.018 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.55 metric 100 2024-02-20T07:50:15.021 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-20T07:50:15.021 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-20T07:50:15.021 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-20T07:50:15.023 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-20T07:50:15.023 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-20T07:50:15.023 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-20T07:50:15.023 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-20T07:50:15.023 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-02-20T07:50:15.024 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-20T07:50:15.024 INFO:teuthology.orchestra.run.smithi055.stdout:Mon IP `172.21.15.55` is in CIDR network `172.21.0.0/20` 2024-02-20T07:50:15.024 INFO:teuthology.orchestra.run.smithi055.stdout:Mon IP `172.21.15.55` is in CIDR network `172.21.0.0/20` 2024-02-20T07:50:15.024 INFO:teuthology.orchestra.run.smithi055.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-20T07:50:15.025 INFO:teuthology.orchestra.run.smithi055.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-20T07:50:15.025 INFO:teuthology.orchestra.run.smithi055.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:50:15.508 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e 2024-02-20T07:50:15.508 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:50:15.509 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Getting image source signatures 2024-02-20T07:50:15.509 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying blob sha256:f8275265412e57f06c462f3ab36d3ba9cb1e55305ee9294fe928d41e45938854 2024-02-20T07:50:15.509 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-20T07:50:15.509 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying config sha256:10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e 2024-02-20T07:50:15.509 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-20T07:50:16.413 INFO:teuthology.orchestra.run.smithi055.stdout:ceph: stdout ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable) 2024-02-20T07:50:16.413 INFO:teuthology.orchestra.run.smithi055.stdout:Ceph version: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable) 2024-02-20T07:50:16.414 INFO:teuthology.orchestra.run.smithi055.stdout:Extracting ceph user uid/gid from container image... 2024-02-20T07:50:17.327 INFO:teuthology.orchestra.run.smithi055.stdout:stat: stdout 167 167 2024-02-20T07:50:17.327 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial keys... 2024-02-20T07:50:18.283 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQC5WdRlcMDAKBAA1qsM/fjyYY5H0E7+yC37Xg== 2024-02-20T07:50:19.214 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQC6WdRlVzMPJBAA/e3rKOcONgnbQumABfeIdw== 2024-02-20T07:50:20.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQC7WdRl+3b5HhAArAvUQXBToxtQ15KyBfTFKw== 2024-02-20T07:50:20.119 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial monmap... 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool for smithi055 [v2:172.21.15.55:3300,v1:172.21.15.55:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = octopus 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: set fsid to 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T07:50:20.975 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:50:20.976 INFO:teuthology.orchestra.run.smithi055.stdout:Creating mon... 2024-02-20T07:50:21.964 INFO:teuthology.orchestra.run.smithi055.stdout:create mon.smithi055 on 2024-02-20T07:50:22.169 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-20T07:50:22.308 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-20T07:50:22.455 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715.target. 2024-02-20T07:50:22.455 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715.target. 2024-02-20T07:50:22.622 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055 2024-02-20T07:50:22.622 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service: Unit ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service not loaded. 2024-02-20T07:50:22.781 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715.target.wants/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service → /etc/systemd/system/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@.service. 2024-02-20T07:50:23.267 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:23 smithi055 podman[28863]: 2024-02-20 07:50:23.201059056 +0000 UTC m=+0.290557198 container start c4663904e2662368562b85d2d2f5bdf14b6a05bdc4d21e8b05d8a41b57efcbfa (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-20T07:50:23.267 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:23 smithi055 ceph-mon[28882]: mkfs 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:23.320 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:50:23.320 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-02-20T07:50:23.327 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:50:23.327 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-02-20T07:50:23.327 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-02-20T07:50:23.327 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to enable service . firewalld.service is not available 2024-02-20T07:50:23.327 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mon to start... 2024-02-20T07:50:23.327 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mon... 2024-02-20T07:50:23.564 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:23 smithi055 ceph-mon[28882]: mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2024-02-20T07:50:23.564 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:23 smithi055 bash[28863]: c4663904e2662368562b85d2d2f5bdf14b6a05bdc4d21e8b05d8a41b57efcbfa 2024-02-20T07:50:23.564 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:23 smithi055 systemd[1]: Started Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:50:24.372 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout cluster: 2024-02-20T07:50:24.372 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout id: 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:24.372 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-20T07:50:24.372 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:24.372 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout services: 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi055 (age 0.573925s) 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout data: 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout pgs: 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e75bd6700 1 Processor -- start 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e75bd6700 1 -- start start 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e75bd6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e70103020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e75bd6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e70103560 con 0x7f0e70102c10 2024-02-20T07:50:24.373 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e6f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e70103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e6f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e70103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43552/0 (socket says 172.21.15.55:43552) 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.747+0000 7f0e6f7fe700 1 -- 172.21.15.55:0/69117316 learned_addr learned my addr 172.21.15.55:0/69117316 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e6f7fe700 1 -- 172.21.15.55:0/69117316 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e701036a0 con 0x7f0e70102c10 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e6f7fe700 1 --2- 172.21.15.55:0/69117316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e70103020 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000bb80 tx=0x7f0e6000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dfe97cbd36e9cc server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e6e7fc700 1 -- 172.21.15.55:0/69117316 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e6000dd00 con 0x7f0e70102c10 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e6e7fc700 1 -- 172.21.15.55:0/69117316 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6000de60 con 0x7f0e70102c10 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e6e7fc700 1 -- 172.21.15.55:0/69117316 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e600105f0 con 0x7f0e70102c10 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e75bd6700 1 -- 172.21.15.55:0/69117316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 msgr2=0x7f0e70103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e75bd6700 1 --2- 172.21.15.55:0/69117316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e70103020 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000bb80 tx=0x7f0e6000b4f0 comp rx=0 tx=0).stop 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e75bd6700 1 -- 172.21.15.55:0/69117316 shutdown_connections 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e75bd6700 1 --2- 172.21.15.55:0/69117316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e70103020 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.748+0000 7f0e75bd6700 1 -- 172.21.15.55:0/69117316 >> 172.21.15.55:0/69117316 conn(0x7f0e700fc580 msgr2=0x7f0e700fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.749+0000 7f0e75bd6700 1 -- 172.21.15.55:0/69117316 shutdown_connections 2024-02-20T07:50:24.374 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.749+0000 7f0e75bd6700 1 -- 172.21.15.55:0/69117316 wait complete. 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.749+0000 7f0e75bd6700 1 Processor -- start 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.749+0000 7f0e75bd6700 1 -- start start 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.749+0000 7f0e75bd6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e7010eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.749+0000 7f0e75bd6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7010f3f0 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e6f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e7010eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e6f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e7010eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43558/0 (socket says 172.21.15.55:43558) 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e6f7fe700 1 -- 172.21.15.55:0/2822277306 learned_addr learned my addr 172.21.15.55:0/2822277306 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e6f7fe700 1 -- 172.21.15.55:0/2822277306 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6000b790 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e6f7fe700 1 --2- 172.21.15.55:0/2822277306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e7010eeb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e60000c00 tx=0x7f0e6001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e600189b0 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6001d070 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7010f5f0 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e60018cd0 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e70110290 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.750+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0e6002e460 con 0x7f0e70102c10 2024-02-20T07:50:24.375 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.751+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0e60014070 con 0x7f0e70102c10 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.751+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7005f5c0 con 0x7f0e70102c10 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.753+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0e6001f050 con 0x7f0e70102c10 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.826+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0e70064ad0 con 0x7f0e70102c10 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.826+0000 7f0e57fff700 1 -- 172.21.15.55:0/2822277306 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f0e60018e30 con 0x7f0e70102c10 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 msgr2=0x7f0e7010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 --2- 172.21.15.55:0/2822277306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e7010eeb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e60000c00 tx=0x7f0e6001a000 comp rx=0 tx=0).stop 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 shutdown_connections 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 --2- 172.21.15.55:0/2822277306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e70102c10 0x7f0e7010eeb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 >> 172.21.15.55:0/2822277306 conn(0x7f0e700fc580 msgr2=0x7f0e70105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 shutdown_connections 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:23.827+0000 7f0e75bd6700 1 -- 172.21.15.55:0/2822277306 wait complete. 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:mon is available 2024-02-20T07:50:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:Assimilating anything we can from ceph.conf... 2024-02-20T07:50:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:24 smithi055 ceph-mon[28882]: mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2024-02-20T07:50:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:24 smithi055 ceph-mon[28882]: monmap e1: 1 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:50:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:24 smithi055 ceph-mon[28882]: fsmap 2024-02-20T07:50:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:24 smithi055 ceph-mon[28882]: osdmap e1: 0 total, 0 up, 0 in 2024-02-20T07:50:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:24 smithi055 ceph-mon[28882]: mgrmap e1: no daemons active 2024-02-20T07:50:24.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:24 smithi055 ceph-mon[28882]: from='client.? 172.21.15.55:0/2822277306' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [global] 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout fsid = 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-20T07:50:25.400 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [osd] 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-20T07:50:25.401 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.797+0000 7f1aaaad0700 1 Processor -- start 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.797+0000 7f1aaaad0700 1 -- start start 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.797+0000 7f1aaaad0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.797+0000 7f1aaaad0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40790f0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa886c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa886c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43572/0 (socket says 172.21.15.55:43572) 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa886c700 1 -- 172.21.15.55:0/3762956151 learned_addr learned my addr 172.21.15.55:0/3762956151 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa886c700 1 -- 172.21.15.55:0/3762956151 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4077730 con 0x7f1aa4078ce0 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa886c700 1 --2- 172.21.15.55:0/3762956151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1a980098e0 tx=0x7f1a98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8a1a97d58829f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa37fe700 1 -- 172.21.15.55:0/3762956151 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9800b860 con 0x7f1aa4078ce0 2024-02-20T07:50:25.402 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa37fe700 1 -- 172.21.15.55:0/3762956151 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9800b9c0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.798+0000 7f1aa37fe700 1 -- 172.21.15.55:0/3762956151 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9800bc90 con 0x7f1aa4078ce0 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 -- 172.21.15.55:0/3762956151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 msgr2=0x7f1aa40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 --2- 172.21.15.55:0/3762956151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1a980098e0 tx=0x7f1a98009250 comp rx=0 tx=0).stop 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 -- 172.21.15.55:0/3762956151 shutdown_connections 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 --2- 172.21.15.55:0/3762956151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 -- 172.21.15.55:0/3762956151 >> 172.21.15.55:0/3762956151 conn(0x7f1aa40fded0 msgr2=0x7f1aa4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 -- 172.21.15.55:0/3762956151 shutdown_connections 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 -- 172.21.15.55:0/3762956151 wait complete. 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 Processor -- start 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.799+0000 7f1aaaad0700 1 -- start start 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aaaad0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa4112f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aaaad0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa41134d0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa886c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa4112f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:25.403 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa886c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa4112f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43574/0 (socket says 172.21.15.55:43574) 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa886c700 1 -- 172.21.15.55:0/1701777603 learned_addr learned my addr 172.21.15.55:0/1701777603 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa886c700 1 -- 172.21.15.55:0/1701777603 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a980094f0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa886c700 1 --2- 172.21.15.55:0/1701777603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa4112f90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009290 tx=0x7f1a98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9801c5c0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9801c740 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa41136d0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.800+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9801ca50 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.801+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa4114370 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.801+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1a9801a450 con 0x7f1aa4078ce0 2024-02-20T07:50:25.404 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.801+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a9802e790 con 0x7f1aa4078ce0 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.802+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa405f5c0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.804+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1a9801ce30 con 0x7f1aa4078ce0 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.873+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1aa410c880 con 0x7f1aa4078ce0 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.876+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1a980243e0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.876+0000 7f1aa17fa700 1 -- 172.21.15.55:0/1701777603 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f1a9802e9f0 con 0x7f1aa4078ce0 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.877+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 msgr2=0x7f1aa4112f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.877+0000 7f1aaaad0700 1 --2- 172.21.15.55:0/1701777603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa4112f90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009290 tx=0x7f1a98016000 comp rx=0 tx=0).stop 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.878+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 shutdown_connections 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.878+0000 7f1aaaad0700 1 --2- 172.21.15.55:0/1701777603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1aa4078ce0 0x7f1aa4112f90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.878+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 >> 172.21.15.55:0/1701777603 conn(0x7f1aa40fded0 msgr2=0x7f1aa4107510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.878+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 shutdown_connections 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:24.878+0000 7f1aaaad0700 1 -- 172.21.15.55:0/1701777603 wait complete. 2024-02-20T07:50:25.405 INFO:teuthology.orchestra.run.smithi055.stdout:Generating new minimal ceph.conf... 2024-02-20T07:50:25.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:25 smithi055 ceph-mon[28882]: from='client.? 172.21.15.55:0/1701777603' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T07:50:25.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:25 smithi055 ceph-mon[28882]: from='client.? 172.21.15.55:0/1701777603' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-20T07:50:26.347 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:26 smithi055 ceph-mon[28882]: from='client.? 172.21.15.55:0/3305448808' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f453b169700 1 Processor -- start 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f453b169700 1 -- start start 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f453b169700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f4534105400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f453b169700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4534105940 con 0x7f4534076ba0 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f4538f05700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f4534105400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f4538f05700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f4534105400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43584/0 (socket says 172.21.15.55:43584) 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f4538f05700 1 -- 172.21.15.55:0/3586230706 learned_addr learned my addr 172.21.15.55:0/3586230706 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.796+0000 7f4538f05700 1 -- 172.21.15.55:0/3586230706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534105a80 con 0x7f4534076ba0 2024-02-20T07:50:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f4538f05700 1 --2- 172.21.15.55:0/3586230706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f4534105400 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f45300098e0 tx=0x7f4530009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af4b026e0e13a5a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f452b7fe700 1 -- 172.21.15.55:0/3586230706 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f453000b8f0 con 0x7f4534076ba0 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f452b7fe700 1 -- 172.21.15.55:0/3586230706 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4530011420 con 0x7f4534076ba0 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f452b7fe700 1 -- 172.21.15.55:0/3586230706 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4530010540 con 0x7f4534076ba0 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f453b169700 1 -- 172.21.15.55:0/3586230706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 msgr2=0x7f4534105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f453b169700 1 --2- 172.21.15.55:0/3586230706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f4534105400 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f45300098e0 tx=0x7f4530009250 comp rx=0 tx=0).stop 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f453b169700 1 -- 172.21.15.55:0/3586230706 shutdown_connections 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f453b169700 1 --2- 172.21.15.55:0/3586230706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f4534105400 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f453b169700 1 -- 172.21.15.55:0/3586230706 >> 172.21.15.55:0/3586230706 conn(0x7f45340fcdd0 msgr2=0x7f45340ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.797+0000 7f453b169700 1 -- 172.21.15.55:0/3586230706 shutdown_connections 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f453b169700 1 -- 172.21.15.55:0/3586230706 wait complete. 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f453b169700 1 Processor -- start 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f453b169700 1 -- start start 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f453b169700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f45341115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:26.355 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f453b169700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4534111b30 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f4538f05700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f45341115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f4538f05700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f45341115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43600/0 (socket says 172.21.15.55:43600) 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.798+0000 7f4538f05700 1 -- 172.21.15.55:0/3305448808 learned_addr learned my addr 172.21.15.55:0/3305448808 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f4538f05700 1 -- 172.21.15.55:0/3305448808 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45300094f0 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f4538f05700 1 --2- 172.21.15.55:0/3305448808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f45341115f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f45300094c0 tx=0x7f4530009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4530010800 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4530010df0 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4534111d30 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4530024e50 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45341129d0 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f453001a6d0 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.799+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f453002d910 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.800+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453405f5c0 con 0x7f4534076ba0 2024-02-20T07:50:26.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.802+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f45300246d0 con 0x7f4534076ba0 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.871+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4534106800 con 0x7f4534076ba0 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.872+0000 7f45297fa700 1 -- 172.21.15.55:0/3305448808 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f4530035090 con 0x7f4534076ba0 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.872+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 msgr2=0x7f45341115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.872+0000 7f453b169700 1 --2- 172.21.15.55:0/3305448808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f45341115f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f45300094c0 tx=0x7f4530009250 comp rx=0 tx=0).stop 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.873+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 shutdown_connections 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.873+0000 7f453b169700 1 --2- 172.21.15.55:0/3305448808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4534076ba0 0x7f45341115f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.873+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 >> 172.21.15.55:0/3305448808 conn(0x7f45340fcdd0 msgr2=0x7f45340fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.873+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 shutdown_connections 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:25.873+0000 7f453b169700 1 -- 172.21.15.55:0/3305448808 wait complete. 2024-02-20T07:50:26.357 INFO:teuthology.orchestra.run.smithi055.stdout:Restarting the monitor... 2024-02-20T07:50:26.635 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:26 smithi055 systemd[1]: Stopping Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715... 2024-02-20T07:50:26.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[28878]: 2024-02-20T07:50:26.632+0000 7efe9fe9b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi055 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T07:50:26.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[28878]: 2024-02-20T07:50:26.632+0000 7efe9fe9b700 -1 mon.smithi055@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T07:50:26.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:26 smithi055 podman[29127]: 2024-02-20 07:50:26.827207785 +0000 UTC m=+0.306114703 container died c4663904e2662368562b85d2d2f5bdf14b6a05bdc4d21e8b05d8a41b57efcbfa (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-20T07:50:27.444 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 podman[29127]: 2024-02-20 07:50:27.191655938 +0000 UTC m=+0.670562862 container cleanup c4663904e2662368562b85d2d2f5bdf14b6a05bdc4d21e8b05d8a41b57efcbfa (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-c80e716, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-20T07:50:27.444 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 bash[29127]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055 2024-02-20T07:50:27.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 podman[29141]: 2024-02-20 07:50:27.442758759 +0000 UTC m=+0.618640903 container remove c4663904e2662368562b85d2d2f5bdf14b6a05bdc4d21e8b05d8a41b57efcbfa (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-20T07:50:27.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service: Deactivated successfully. 2024-02-20T07:50:27.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 systemd[1]: Stopped Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:50:27.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 systemd[1]: Starting Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715... 2024-02-20T07:50:28.000 INFO:teuthology.orchestra.run.smithi055.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-20T07:50:28.023 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 podman[29235]: 2024-02-20 07:50:27.760701306 +0000 UTC m=+0.095877315 container create a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 podman[29235]: 2024-02-20 07:50:27.682348676 +0000 UTC m=+0.017524650 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph:quincy 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 podman[29235]: 2024-02-20 07:50:27.921425119 +0000 UTC m=+0.256601086 container init a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 podman[29235]: 2024-02-20 07:50:27.924224973 +0000 UTC m=+0.259400947 container start a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: set uid:gid to 167:167 (ceph:ceph) 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable), process ceph-mon, pid 2 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: pidfile_write: ignore empty --pid-file 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: load: jerasure load: lrc 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: RocksDB version: 6.15.5 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Compile date Feb 19 2024 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: DB SUMMARY 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: DB Session ID: E50WULVU5KEA5OF2WU28 2024-02-20T07:50:28.024 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: CURRENT file: CURRENT 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: IDENTITY file: IDENTITY 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi055/store.db dir, Total Num: 1, files: 000008.sst 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi055/store.db: 000010.log size: 97282 ; 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.error_if_exists: 0 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.create_if_missing: 0 2024-02-20T07:50:28.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.paranoid_checks: 1 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.env: 0x55f13d8f9a80 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.fs: Posix File System 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.info_log: 0x55f13fc13cc0 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_file_opening_threads: 16 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.statistics: (nil) 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.use_fsync: 0 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_log_file_size: 0 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T07:50:28.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.keep_log_file_num: 1000 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.recycle_log_file_num: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.allow_fallocate: 1 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.allow_mmap_reads: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.allow_mmap_writes: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.use_direct_reads: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.create_missing_column_families: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.db_log_dir: 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi055/store.db 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T07:50:28.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.advise_random_on_open: 1 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.db_write_buffer_size: 0 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.write_buffer_manager: 0x55f140988b40 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.rate_limiter: (nil) 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.wal_recovery_mode: 2 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.enable_thread_tracking: 0 2024-02-20T07:50:28.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.enable_pipelined_write: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.unordered_write: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.row_cache: None 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.wal_filter: None 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.allow_ingest_behind: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.preserve_deletes: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.two_write_queues: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.manual_wal_flush: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.atomic_flush: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T07:50:28.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.log_readahead_size: 0 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.best_efforts_recovery: 0 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.allow_data_in_errors: 0 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.db_host_id: __hostname__ 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_background_jobs: 2 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_background_compactions: -1 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_subcompactions: 1 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T07:50:28.031 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_total_wal_size: 0 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_open_files: -1 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bytes_per_sync: 0 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_readahead_size: 0 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_background_flushes: -1 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Compression algorithms supported: 2024-02-20T07:50:28.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kZSTD supported: 0 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kXpressCompression supported: 0 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kLZ4HCCompression supported: 1 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kZlibCompression supported: 1 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kSnappyCompression supported: 1 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kLZ4Compression supported: 1 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: kBZip2Compression supported: 0 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000009 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.merge_operator: 2024-02-20T07:50:28.033 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_filter: None 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_filter_factory: None 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.sst_partitioner_factory: None 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f13fc68f58) 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: cache_index_and_filter_blocks: 1 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: pin_top_level_index_and_filter: 1 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: index_type: 0 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: data_block_index_type: 0 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: index_shortening: 1 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: hash_index_allow_collision: 1 2024-02-20T07:50:28.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: checksum: 1 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: no_block_cache: 0 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache: 0x55f13fc9b090 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache_name: BinnedLRUCache 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache_options: 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: capacity : 536870912 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: num_shard_bits : 4 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: strict_capacity_limit : 0 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: high_pri_pool_ratio: 0.000 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache_compressed: (nil) 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: persistent_cache: (nil) 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_size: 4096 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_size_deviation: 10 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_restart_interval: 16 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: index_block_restart_interval: 1 2024-02-20T07:50:28.035 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: metadata_block_size: 4096 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: partition_filters: 0 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: use_delta_encoding: 1 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: whole_key_filtering: 1 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: verify_compression: 0 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: read_amp_bytes_per_bit: 0 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: format_version: 4 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: enable_index_compression: 1 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_align: 0 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.write_buffer_size: 33554432 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_write_buffer_number: 2 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression: NoCompression 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression: Disabled 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.prefix_extractor: nullptr 2024-02-20T07:50:28.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.num_levels: 7 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.level: 32767 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.strategy: 0 2024-02-20T07:50:28.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compression_opts.enabled: false 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.target_file_size_base: 67108864 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T07:50:28.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.arena_block_size: 4194304 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T07:50:28.039 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.disable_auto_compactions: 0 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T07:50:28.040 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.table_properties_collectors: 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.inplace_update_support: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.bloom_locality: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.max_successive_merges: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.paranoid_file_checks: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.force_consistency_checks: 1 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.report_bg_io_stats: 0 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.ttl: 2592000 2024-02-20T07:50:28.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.enable_blob_files: false 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.min_blob_size: 0 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.blob_file_size: 268435456 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415427950014, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415427952003, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92554, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90649, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83364, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708415427, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "819c664a-8cf1-4453-976d-daaefe4c29ae", "db_session_id": "E50WULVU5KEA5OF2WU28"}} 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-20T07:50:28.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415427952780, "job": 1, "event": "recovery_finished"} 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi055/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55f13fc2f180 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: DB pointer 0x55f13fd04000 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** DB Stats ** 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T07:50:28.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T07:50:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative compaction: 0.00 GB write, 16.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval compaction: 0.00 GB write, 16.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 50.6 0.00 0.00 1 0.002 0 0 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative compaction: 0.00 GB write, 16.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:50:28.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: starting mon.smithi055 rank 0 at public addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] at bind addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi055 fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???) e1 preinit fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).mds e1 new map 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).mds e1 print_map 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: e1 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: legacy client fscid: -1 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: No filesystems configured 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:50:28.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 ceph-mon[29254]: mon.smithi055@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 bash[29235]: a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:27 smithi055 systemd[1]: Started Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:28 smithi055 ceph-mon[29254]: mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:28 smithi055 ceph-mon[29254]: monmap e1: 1 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:28 smithi055 ceph-mon[29254]: fsmap 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:28 smithi055 ceph-mon[29254]: osdmap e1: 0 total, 0 up, 0 in 2024-02-20T07:50:28.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:28 smithi055 ceph-mon[29254]: mgrmap e1: no daemons active 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.449+0000 7ff28f04f700 1 Processor -- start 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.449+0000 7ff28f04f700 1 -- start start 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.449+0000 7ff28f04f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.449+0000 7ff28f04f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288101fc0 con 0x7ff288101670 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff28cdeb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff28cdeb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43606/0 (socket says 172.21.15.55:43606) 2024-02-20T07:50:29.053 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff28cdeb700 1 -- 172.21.15.55:0/547954518 learned_addr learned my addr 172.21.15.55:0/547954518 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff28cdeb700 1 -- 172.21.15.55:0/547954518 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff288102100 con 0x7ff288101670 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff28cdeb700 1 --2- 172.21.15.55:0/547954518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288101a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0098e0 tx=0x7ff27c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef0dad7f19fa214 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff2877fe700 1 -- 172.21.15.55:0/547954518 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff27c00b860 con 0x7ff288101670 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff2877fe700 1 -- 172.21.15.55:0/547954518 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff27c00be50 con 0x7ff288101670 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.450+0000 7ff2877fe700 1 -- 172.21.15.55:0/547954518 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff27c010380 con 0x7ff288101670 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 -- 172.21.15.55:0/547954518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 msgr2=0x7ff288101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 --2- 172.21.15.55:0/547954518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288101a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0098e0 tx=0x7ff27c009250 comp rx=0 tx=0).stop 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 -- 172.21.15.55:0/547954518 shutdown_connections 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 --2- 172.21.15.55:0/547954518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288101a80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 -- 172.21.15.55:0/547954518 >> 172.21.15.55:0/547954518 conn(0x7ff2880fcdd0 msgr2=0x7ff2880ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 -- 172.21.15.55:0/547954518 shutdown_connections 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 -- 172.21.15.55:0/547954518 wait complete. 2024-02-20T07:50:29.054 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 Processor -- start 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.451+0000 7ff28f04f700 1 -- start start 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28f04f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288110740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28f04f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288110c80 con 0x7ff288101670 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28cdeb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288110740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28cdeb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288110740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43616/0 (socket says 172.21.15.55:43616) 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28cdeb700 1 -- 172.21.15.55:0/3983148748 learned_addr learned my addr 172.21.15.55:0/3983148748 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28cdeb700 1 -- 172.21.15.55:0/3983148748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c0094f0 con 0x7ff288101670 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff28cdeb700 1 --2- 172.21.15.55:0/3983148748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288110740 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00b440 tx=0x7ff27c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.452+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff27c010760 con 0x7ff288101670 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.453+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff27c010d50 con 0x7ff288101670 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.453+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288110e80 con 0x7ff288101670 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.453+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288111b20 con 0x7ff288101670 2024-02-20T07:50:29.055 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.453+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff27c024d80 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.453+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff27c01a6b0 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.453+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff27c02d900 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.454+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28805f5c0 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.456+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff27c0108c0 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.524+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ff28804bed0 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.528+0000 7ff2857fa700 1 -- 172.21.15.55:0/3983148748 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff27c010a60 con 0x7ff288101670 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 msgr2=0x7ff288110740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 --2- 172.21.15.55:0/3983148748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288110740 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00b440 tx=0x7ff27c009250 comp rx=0 tx=0).stop 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 shutdown_connections 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 --2- 172.21.15.55:0/3983148748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff288101670 0x7ff288110740 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 >> 172.21.15.55:0/3983148748 conn(0x7ff2880fcdd0 msgr2=0x7ff2880fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 shutdown_connections 2024-02-20T07:50:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:28.529+0000 7ff28f04f700 1 -- 172.21.15.55:0/3983148748 wait complete. 2024-02-20T07:50:29.057 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-20T07:50:29.057 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-20T07:50:29.057 INFO:teuthology.orchestra.run.smithi055.stdout:Creating mgr... 2024-02-20T07:50:29.057 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying port 9283 ... 2024-02-20T07:50:29.224 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mgr.smithi055.stjper 2024-02-20T07:50:29.224 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mgr.smithi055.stjper.service: Unit ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mgr.smithi055.stjper.service not loaded. 2024-02-20T07:50:29.370 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715.target.wants/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mgr.smithi055.stjper.service → /etc/systemd/system/ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@.service. 2024-02-20T07:50:29.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:29 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3983148748' entity='client.admin' 2024-02-20T07:50:29.973 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:50:29.973 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-02-20T07:50:29.981 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:50:29.981 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-02-20T07:50:29.981 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-02-20T07:50:29.981 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to enable service . firewalld.service is not available 2024-02-20T07:50:29.989 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:50:29.989 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-02-20T07:50:29.997 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:50:29.997 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-02-20T07:50:29.997 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-02-20T07:50:29.997 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-20T07:50:29.997 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr to start... 2024-02-20T07:50:29.997 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr... 2024-02-20T07:50:30.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:30 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1506703956' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:50:31.047 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:31.047 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:50:31.047 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "9a1162a4-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:50:31.048 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:50:31.049 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:50:31.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:50:31.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:50:23.257654+0000", 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:31.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.401+0000 7eff479d0700 1 Processor -- start 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.401+0000 7eff479d0700 1 -- start start 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.401+0000 7eff479d0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.401+0000 7eff479d0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff401030c0 con 0x7eff40102770 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff4576c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff4576c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43634/0 (socket says 172.21.15.55:43634) 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff4576c700 1 -- 172.21.15.55:0/1872402268 learned_addr learned my addr 172.21.15.55:0/1872402268 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff4576c700 1 -- 172.21.15.55:0/1872402268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff40103200 con 0x7eff40102770 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff4576c700 1 --2- 172.21.15.55:0/1872402268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40102b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098e0 tx=0x7eff3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36648ddb2de75fa server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff33fff700 1 -- 172.21.15.55:0/1872402268 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c00b860 con 0x7eff40102770 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff33fff700 1 -- 172.21.15.55:0/1872402268 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7eff3c00be50 con 0x7eff40102770 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.402+0000 7eff33fff700 1 -- 172.21.15.55:0/1872402268 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c010380 con 0x7eff40102770 2024-02-20T07:50:31.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 -- 172.21.15.55:0/1872402268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 msgr2=0x7eff40102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 --2- 172.21.15.55:0/1872402268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40102b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098e0 tx=0x7eff3c009250 comp rx=0 tx=0).stop 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 -- 172.21.15.55:0/1872402268 shutdown_connections 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 --2- 172.21.15.55:0/1872402268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40102b80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 -- 172.21.15.55:0/1872402268 >> 172.21.15.55:0/1872402268 conn(0x7eff400fded0 msgr2=0x7eff40100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 -- 172.21.15.55:0/1872402268 shutdown_connections 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.403+0000 7eff479d0700 1 -- 172.21.15.55:0/1872402268 wait complete. 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff479d0700 1 Processor -- start 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff479d0700 1 -- start start 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff479d0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff479d0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff40111aa0 con 0x7eff40102770 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff4576c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff4576c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43648/0 (socket says 172.21.15.55:43648) 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff4576c700 1 -- 172.21.15.55:0/1506703956 learned_addr learned my addr 172.21.15.55:0/1506703956 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:31.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff4576c700 1 -- 172.21.15.55:0/1506703956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0094f0 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff4576c700 1 --2- 172.21.15.55:0/1506703956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40111560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00b440 tx=0x7eff3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.404+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c010760 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.405+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff40111ca0 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.405+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7eff3c010d50 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.405+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c01ae20 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.405+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff40112940 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.405+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7eff3c02e6c0 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.405+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7eff3c024900 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.406+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4005f5c0 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.408+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7eff3c029070 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.481+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7eff401045e0 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.481+0000 7eff31ffb700 1 -- 172.21.15.55:0/1506703956 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7eff3c01a6a0 con 0x7eff40102770 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 msgr2=0x7eff40111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 --2- 172.21.15.55:0/1506703956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40111560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00b440 tx=0x7eff3c009250 comp rx=0 tx=0).stop 2024-02-20T07:50:31.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 shutdown_connections 2024-02-20T07:50:31.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 --2- 172.21.15.55:0/1506703956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff40102770 0x7eff40111560 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:31.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 >> 172.21.15.55:0/1506703956 conn(0x7eff400fded0 msgr2=0x7eff400ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:31.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 shutdown_connections 2024-02-20T07:50:31.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:30.482+0000 7eff479d0700 1 -- 172.21.15.55:0/1506703956 wait complete. 2024-02-20T07:50:31.063 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (1/15)... 2024-02-20T07:50:34.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:33 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2657811780' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:50:36.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: Activating manager daemon smithi055.stjper 2024-02-20T07:50:36.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: mgrmap e2: smithi055.stjper(active, starting, since 0.00320747s) 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr metadata", "who": "smithi055.stjper", "id": "smithi055.stjper"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: Manager daemon smithi055.stjper is now available 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/trash_purge_schedule"}]: dispatch 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' 2024-02-20T07:50:36.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:36 smithi055 ceph-mon[29254]: from='mgr.14100 172.21.15.55:0/1858549336' entity='mgr.smithi055.stjper' 2024-02-20T07:50:37.115 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:37.117 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:50:37.117 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "9a1162a4-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:50:37.117 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:37.118 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:50:37.119 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:50:37.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:50:37.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:50:37.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:50:37.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:50:37.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:50:37.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:50:37.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:37.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:50:23.257654+0000", 2024-02-20T07:50:37.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed2aad700 1 Processor -- start 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed2aad700 1 -- start start 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed2aad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed2aad700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc079110 con 0x7f8ecc078d00 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed0849700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed0849700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55284/0 (socket says 172.21.15.55:55284) 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.448+0000 7f8ed0849700 1 -- 172.21.15.55:0/433900088 learned_addr learned my addr 172.21.15.55:0/433900088 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.449+0000 7f8ed0849700 1 -- 172.21.15.55:0/433900088 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc077750 con 0x7f8ecc078d00 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.449+0000 7f8ed0849700 1 --2- 172.21.15.55:0/433900088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc077210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00098e0 tx=0x7f8ec0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c004f4748a899ca1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:37.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.449+0000 7f8ecb7fe700 1 -- 172.21.15.55:0/433900088 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec000b860 con 0x7f8ecc078d00 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.449+0000 7f8ecb7fe700 1 -- 172.21.15.55:0/433900088 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8ec000be50 con 0x7f8ecc078d00 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.449+0000 7f8ecb7fe700 1 -- 172.21.15.55:0/433900088 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec0010380 con 0x7f8ecc078d00 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 -- 172.21.15.55:0/433900088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 msgr2=0x7f8ecc077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 --2- 172.21.15.55:0/433900088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc077210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00098e0 tx=0x7f8ec0009250 comp rx=0 tx=0).stop 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 -- 172.21.15.55:0/433900088 shutdown_connections 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 --2- 172.21.15.55:0/433900088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc077210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 -- 172.21.15.55:0/433900088 >> 172.21.15.55:0/433900088 conn(0x7f8ecc0fdf00 msgr2=0x7f8ecc100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 -- 172.21.15.55:0/433900088 shutdown_connections 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 -- 172.21.15.55:0/433900088 wait complete. 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 Processor -- start 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.450+0000 7f8ed2aad700 1 -- start start 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed2aad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc1137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed2aad700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc113cf0 con 0x7f8ecc078d00 2024-02-20T07:50:37.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed0849700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc1137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed0849700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc1137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55298/0 (socket says 172.21.15.55:55298) 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed0849700 1 -- 172.21.15.55:0/2657811780 learned_addr learned my addr 172.21.15.55:0/2657811780 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed0849700 1 -- 172.21.15.55:0/2657811780 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00094f0 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed0849700 1 --2- 172.21.15.55:0/2657811780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc1137b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0006b00 tx=0x7f8ec0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec00107c0 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8ec0010db0 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc113ef0 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec001acf0 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.451+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc114b90 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.452+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8ec0024700 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.452+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8ec0019b40 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.452+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc05f5c0 con 0x7f8ecc078d00 2024-02-20T07:50:37.134 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.455+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8ec001a530 con 0x7f8ecc078d00 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.526+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8ecc108a80 con 0x7f8ecc078d00 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.527+0000 7f8ec97fa700 1 -- 172.21.15.55:0/2657811780 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f8ec0019d00 con 0x7f8ecc078d00 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.527+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 msgr2=0x7f8ecc1137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.527+0000 7f8ed2aad700 1 --2- 172.21.15.55:0/2657811780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc1137b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0006b00 tx=0x7f8ec0009250 comp rx=0 tx=0).stop 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.528+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 shutdown_connections 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.528+0000 7f8ed2aad700 1 --2- 172.21.15.55:0/2657811780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ecc078d00 0x7f8ecc1137b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.528+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 >> 172.21.15.55:0/2657811780 conn(0x7f8ecc0fdf00 msgr2=0x7f8ecc0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.528+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 shutdown_connections 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:33.528+0000 7f8ed2aad700 1 -- 172.21.15.55:0/2657811780 wait complete. 2024-02-20T07:50:37.135 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (2/15)... 2024-02-20T07:50:38.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:38 smithi055 ceph-mon[29254]: mgrmap e3: smithi055.stjper(active, since 1.00767s) 2024-02-20T07:50:40.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:40 smithi055 ceph-mon[29254]: mgrmap e4: smithi055.stjper(active, since 3s) 2024-02-20T07:50:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:43 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/61504335' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:50:44.895 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "9a1162a4-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:50:44.896 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:50:44.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:50:44.906 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:50:44.907 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:50:23.257654+0000", 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:50:44.908 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2849e700 1 Processor -- start 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2849e700 1 -- start start 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2849e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2849e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f201030c0 con 0x7f0f20102770 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2623a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2623a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57014/0 (socket says 172.21.15.55:57014) 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.799+0000 7f0f2623a700 1 -- 172.21.15.55:0/1669792072 learned_addr learned my addr 172.21.15.55:0/1669792072 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f2623a700 1 -- 172.21.15.55:0/1669792072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f20103200 con 0x7f0f20102770 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f2623a700 1 --2- 172.21.15.55:0/1669792072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20102b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0098e0 tx=0x7f0f1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8948e6f7fd46040d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f25238700 1 -- 172.21.15.55:0/1669792072 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c00b860 con 0x7f0f20102770 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f25238700 1 -- 172.21.15.55:0/1669792072 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0f1c00be50 con 0x7f0f20102770 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f25238700 1 -- 172.21.15.55:0/1669792072 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c010380 con 0x7f0f20102770 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f2849e700 1 -- 172.21.15.55:0/1669792072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 msgr2=0x7f0f20102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:44.909 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.800+0000 7f0f2849e700 1 --2- 172.21.15.55:0/1669792072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20102b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0098e0 tx=0x7f0f1c009250 comp rx=0 tx=0).stop 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 -- 172.21.15.55:0/1669792072 shutdown_connections 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 --2- 172.21.15.55:0/1669792072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20102b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 -- 172.21.15.55:0/1669792072 >> 172.21.15.55:0/1669792072 conn(0x7f0f200fded0 msgr2=0x7f0f20100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 -- 172.21.15.55:0/1669792072 shutdown_connections 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 -- 172.21.15.55:0/1669792072 wait complete. 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 Processor -- start 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 -- start start 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.801+0000 7f0f2849e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f20111aa0 con 0x7f0f20102770 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2623a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2623a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57026/0 (socket says 172.21.15.55:57026) 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2623a700 1 -- 172.21.15.55:0/61504335 learned_addr learned my addr 172.21.15.55:0/61504335 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2623a700 1 -- 172.21.15.55:0/61504335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0094f0 con 0x7f0f20102770 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2623a700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00b440 tx=0x7f0f1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:44.910 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c010760 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f20111ca0 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0f1c010d50 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c01ae20 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.802+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f20112940 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.803+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f0f1c0108c0 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.803+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2005f5c0 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.803+0000 7f0f16ffd700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f0f0c036720 0x7f0f0c038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.803+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0f1c04a550 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.803+0000 7f0f25a39700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f0f0c036720 0x7f0f0c038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.804+0000 7f0f25a39700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f0f0c036720 0x7f0f0c038bd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009990 tx=0x7f0f10006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:42.808+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0f1c018b40 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.081+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0f2004bed0 con 0x7f0f20102770 2024-02-20T07:50:44.911 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.081+0000 7f0f16ffd700 1 -- 172.21.15.55:0/61504335 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f0f1c018b40 con 0x7f0f20102770 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f0f0c036720 msgr2=0x7f0f0c038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f0f0c036720 0x7f0f0c038bd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009990 tx=0x7f0f10006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 msgr2=0x7f0f20111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00b440 tx=0x7f0f1c009250 comp rx=0 tx=0).stop 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 shutdown_connections 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f0f0c036720 0x7f0f0c038bd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 --2- 172.21.15.55:0/61504335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f20102770 0x7f0f20111560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.083+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 >> 172.21.15.55:0/61504335 conn(0x7f0f200fded0 msgr2=0x7f0f200ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.084+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 shutdown_connections 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:43.084+0000 7f0f2849e700 1 -- 172.21.15.55:0/61504335 wait complete. 2024-02-20T07:50:44.912 INFO:teuthology.orchestra.run.smithi055.stdout:mgr is available 2024-02-20T07:50:45.889 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:45 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/25588035' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T07:50:46.149 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:46.149 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [global] 2024-02-20T07:50:46.149 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout fsid = 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:46.149 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-20T07:50:46.149 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-20T07:50:46.149 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [osd] 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14f140e700 1 Processor -- start 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14f140e700 1 -- start start 2024-02-20T07:50:46.150 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14f140e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec104270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14f140e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1047b0 con 0x7f14ec103e60 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14eaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec104270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14eaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec104270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57034/0 (socket says 172.21.15.55:57034) 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.389+0000 7f14eaffd700 1 -- 172.21.15.55:0/2659019439 learned_addr learned my addr 172.21.15.55:0/2659019439 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14eaffd700 1 -- 172.21.15.55:0/2659019439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec1048f0 con 0x7f14ec103e60 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14eaffd700 1 --2- 172.21.15.55:0/2659019439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec104270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009e50 tx=0x7f14e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a6a07ab44bc4e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14e9ffb700 1 -- 172.21.15.55:0/2659019439 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e40092e0 con 0x7f14ec103e60 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14e9ffb700 1 -- 172.21.15.55:0/2659019439 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f14e4012430 con 0x7f14ec103e60 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14e9ffb700 1 -- 172.21.15.55:0/2659019439 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e40075c0 con 0x7f14ec103e60 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14f140e700 1 -- 172.21.15.55:0/2659019439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 msgr2=0x7f14ec104270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.390+0000 7f14f140e700 1 --2- 172.21.15.55:0/2659019439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec104270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009e50 tx=0x7f14e4009840 comp rx=0 tx=0).stop 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 -- 172.21.15.55:0/2659019439 shutdown_connections 2024-02-20T07:50:46.151 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 --2- 172.21.15.55:0/2659019439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec104270 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 -- 172.21.15.55:0/2659019439 >> 172.21.15.55:0/2659019439 conn(0x7f14ec0fc580 msgr2=0x7f14ec0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 -- 172.21.15.55:0/2659019439 shutdown_connections 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 -- 172.21.15.55:0/2659019439 wait complete. 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 Processor -- start 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.391+0000 7f14f140e700 1 -- start start 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14f140e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec1036a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14f140e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec101b20 con 0x7f14ec103e60 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14eaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec1036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14eaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec1036a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57050/0 (socket says 172.21.15.55:57050) 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14eaffd700 1 -- 172.21.15.55:0/25588035 learned_addr learned my addr 172.21.15.55:0/25588035 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14eaffd700 1 -- 172.21.15.55:0/25588035 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e4009ae0 con 0x7f14ec103e60 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14eaffd700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec1036a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f14e4000f80 tx=0x7f14e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e40092e0 con 0x7f14ec103e60 2024-02-20T07:50:46.152 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f14e4007cc0 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec101d20 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e401c990 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.392+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec102210 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.393+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f14e4019070 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.393+0000 7f14e37fe700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f14d8036790 0x7f14d8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.393+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f14e4049dd0 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.394+0000 7f14ea7fc700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f14d8036790 0x7f14d8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.394+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec10b640 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.394+0000 7f14ea7fc700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f14d8036790 0x7f14d8038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009990 tx=0x7f14d4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.398+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f14e4017080 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.589+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f14ec05f5c0 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.591+0000 7f14e37fe700 1 -- 172.21.15.55:0/25588035 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f14e4043e60 con 0x7f14ec103e60 2024-02-20T07:50:46.153 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.592+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f14d8036790 msgr2=0x7f14d8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.592+0000 7f14f140e700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f14d8036790 0x7f14d8038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009990 tx=0x7f14d4006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.592+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 msgr2=0x7f14ec1036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.592+0000 7f14f140e700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec1036a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f14e4000f80 tx=0x7f14e4009840 comp rx=0 tx=0).stop 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.593+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 shutdown_connections 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.593+0000 7f14f140e700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f14d8036790 0x7f14d8038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.593+0000 7f14f140e700 1 --2- 172.21.15.55:0/25588035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14ec103e60 0x7f14ec1036a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.593+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 >> 172.21.15.55:0/25588035 conn(0x7f14ec0fc580 msgr2=0x7f14ec0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.593+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 shutdown_connections 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:45.593+0000 7f14f140e700 1 -- 172.21.15.55:0/25588035 wait complete. 2024-02-20T07:50:46.154 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling cephadm module... 2024-02-20T07:50:47.163 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:46 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3894128533' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-20T07:50:47.438 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.572+0000 7efeef2a8700 1 Processor -- start 2024-02-20T07:50:47.438 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.572+0000 7efeef2a8700 1 -- start start 2024-02-20T07:50:47.438 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.572+0000 7efeef2a8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.572+0000 7efeef2a8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee8101fc0 con 0x7efee8101670 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efeed044700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efeed044700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57066/0 (socket says 172.21.15.55:57066) 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efeed044700 1 -- 172.21.15.55:0/1915791010 learned_addr learned my addr 172.21.15.55:0/1915791010 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efeed044700 1 -- 172.21.15.55:0/1915791010 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8102100 con 0x7efee8101670 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efeed044700 1 --2- 172.21.15.55:0/1915791010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8101a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efee40098e0 tx=0x7efee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d1db97bc08fd78 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efedffff700 1 -- 172.21.15.55:0/1915791010 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee400b860 con 0x7efee8101670 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efedffff700 1 -- 172.21.15.55:0/1915791010 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7efee400be50 con 0x7efee8101670 2024-02-20T07:50:47.439 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.573+0000 7efedffff700 1 -- 172.21.15.55:0/1915791010 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee4010380 con 0x7efee8101670 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 -- 172.21.15.55:0/1915791010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 msgr2=0x7efee8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 --2- 172.21.15.55:0/1915791010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8101a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efee40098e0 tx=0x7efee4009250 comp rx=0 tx=0).stop 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 -- 172.21.15.55:0/1915791010 shutdown_connections 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 --2- 172.21.15.55:0/1915791010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8101a80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 -- 172.21.15.55:0/1915791010 >> 172.21.15.55:0/1915791010 conn(0x7efee80fcdd0 msgr2=0x7efee80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 -- 172.21.15.55:0/1915791010 shutdown_connections 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 -- 172.21.15.55:0/1915791010 wait complete. 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.574+0000 7efeef2a8700 1 Processor -- start 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeef2a8700 1 -- start start 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeef2a8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8114240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeef2a8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee8076bf0 con 0x7efee8101670 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeed044700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8114240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:47.440 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeed044700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8114240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57068/0 (socket says 172.21.15.55:57068) 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeed044700 1 -- 172.21.15.55:0/3894128533 learned_addr learned my addr 172.21.15.55:0/3894128533 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeed044700 1 -- 172.21.15.55:0/3894128533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40094f0 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeed044700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8114240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efee400b440 tx=0x7efee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee4010760 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7efee4010d50 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee4024d80 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.575+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee8076df0 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.576+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8114940 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.576+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7efee40108c0 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.577+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee804bed0 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.577+0000 7efeddffb700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7efed40366e0 0x7efed4038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.577+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efee404a5d0 con 0x7efee8101670 2024-02-20T07:50:47.441 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.577+0000 7efeec843700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7efed40366e0 0x7efed4038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.577+0000 7efeec843700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7efed40366e0 0x7efed4038b90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efed8009990 tx=0x7efed8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.582+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7efee4010b70 con 0x7efee8101670 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.830+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7efee8114b90 con 0x7efee8101670 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.883+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7efee40233a0 con 0x7efee8101670 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.884+0000 7efeddffb700 1 -- 172.21.15.55:0/3894128533 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7efee401a430 con 0x7efee8101670 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7efed40366e0 msgr2=0x7efed4038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7efed40366e0 0x7efed4038b90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efed8009990 tx=0x7efed8006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 msgr2=0x7efee8114240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8114240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efee400b440 tx=0x7efee4009250 comp rx=0 tx=0).stop 2024-02-20T07:50:47.442 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 shutdown_connections 2024-02-20T07:50:47.443 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7efed40366e0 0x7efed4038b90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:47.443 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 --2- 172.21.15.55:0/3894128533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee8101670 0x7efee8114240 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:47.443 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 >> 172.21.15.55:0/3894128533 conn(0x7efee80fcdd0 msgr2=0x7efee80fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:47.443 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 shutdown_connections 2024-02-20T07:50:47.443 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:46.887+0000 7efeef2a8700 1 -- 172.21.15.55:0/3894128533 wait complete. 2024-02-20T07:50:48.052 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:47 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3894128533' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-20T07:50:48.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:47 smithi055 ceph-mon[29254]: mgrmap e5: smithi055.stjper(active, since 10s) 2024-02-20T07:50:48.654 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:50:48.654 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-20T07:50:48.654 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "active_name": "smithi055.stjper", 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3cf5757700 1 Processor -- start 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3cf5757700 1 -- start start 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3cf5757700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3cf5757700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0079130 con 0x7f3cf0078d20 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57090/0 (socket says 172.21.15.55:57090) 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3ceeffd700 1 -- 172.21.15.55:0/3072833097 learned_addr learned my addr 172.21.15.55:0/3072833097 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.845+0000 7f3ceeffd700 1 -- 172.21.15.55:0/3072833097 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0077770 con 0x7f3cf0078d20 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.846+0000 7f3ceeffd700 1 --2- 172.21.15.55:0/3072833097 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0077230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00098e0 tx=0x7f3ce0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9f251e2c0016c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.846+0000 7f3cedffb700 1 -- 172.21.15.55:0/3072833097 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce000b860 con 0x7f3cf0078d20 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.846+0000 7f3cedffb700 1 -- 172.21.15.55:0/3072833097 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3ce000be50 con 0x7f3cf0078d20 2024-02-20T07:50:48.655 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.846+0000 7f3cedffb700 1 -- 172.21.15.55:0/3072833097 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce0010380 con 0x7f3cf0078d20 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.846+0000 7f3cf5757700 1 -- 172.21.15.55:0/3072833097 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 msgr2=0x7f3cf0077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 --2- 172.21.15.55:0/3072833097 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0077230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00098e0 tx=0x7f3ce0009250 comp rx=0 tx=0).stop 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 -- 172.21.15.55:0/3072833097 shutdown_connections 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 --2- 172.21.15.55:0/3072833097 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0077230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 -- 172.21.15.55:0/3072833097 >> 172.21.15.55:0/3072833097 conn(0x7f3cf00fdf00 msgr2=0x7f3cf0100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 -- 172.21.15.55:0/3072833097 shutdown_connections 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 -- 172.21.15.55:0/3072833097 wait complete. 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 Processor -- start 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.847+0000 7f3cf5757700 1 -- start start 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cf5757700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cf5757700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0118120 con 0x7f3cf0078d20 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:48.656 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57106/0 (socket says 172.21.15.55:57106) 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3ceeffd700 1 -- 172.21.15.55:0/1619618368 learned_addr learned my addr 172.21.15.55:0/1619618368 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3ceeffd700 1 -- 172.21.15.55:0/1619618368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00094f0 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3ceeffd700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0117be0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0006fd0 tx=0x7f3ce0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce0010760 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3ce0010d50 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf0118320 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce001ae10 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.848+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf0118fc0 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.849+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf005f5c0 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.849+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3ce00193c0 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.850+0000 7f3cd77fe700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 0x7f3cd8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.850+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3ce0049f40 con 0x7f3cf0078d20 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.850+0000 7f3cee7fc700 1 -- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 msgr2=0x7f3cd8038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.850+0000 7f3cee7fc700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 0x7f3cd8038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:50:48.657 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:47.854+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3ce0019d20 con 0x7f3cf0078d20 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.051+0000 7f3cee7fc700 1 -- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 msgr2=0x7f3cd8038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.051+0000 7f3cee7fc700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 0x7f3cd8038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.101+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3cf0119ad0 con 0x7f3cf0078d20 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.102+0000 7f3cd77fe700 1 -- 172.21.15.55:0/1619618368 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3ce002b300 con 0x7f3cf0078d20 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 msgr2=0x7f3cd8038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 0x7f3cd8038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 msgr2=0x7f3cf0117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0117be0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0006fd0 tx=0x7f3ce0009250 comp rx=0 tx=0).stop 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 shutdown_connections 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f3cd8036790 0x7f3cd8038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 --2- 172.21.15.55:0/1619618368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0078d20 0x7f3cf0117be0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 >> 172.21.15.55:0/1619618368 conn(0x7f3cf00fdf00 msgr2=0x7f3cf01002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 shutdown_connections 2024-02-20T07:50:48.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:48.104+0000 7f3cf5757700 1 -- 172.21.15.55:0/1619618368 wait complete. 2024-02-20T07:50:48.659 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for the mgr to restart... 2024-02-20T07:50:48.659 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr epoch 5... 2024-02-20T07:50:49.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:48 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1619618368' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T07:50:54.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: Active manager daemon smithi055.stjper restarted 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: Activating manager daemon smithi055.stjper 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: osdmap e2: 0 total, 0 up, 0 in 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: mgrmap e6: smithi055.stjper(active, starting, since 0.00846045s) 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr metadata", "who": "smithi055.stjper", "id": "smithi055.stjper"}]: dispatch 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: Manager daemon smithi055.stjper is now available 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: Found migration_current of "None". Setting to last migration. 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:54.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:54.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:50:54.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:53 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/trash_purge_schedule"}]: dispatch 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d42be700 1 Processor -- start 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d42be700 1 -- start start 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d42be700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d42be700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc1030c0 con 0x7f58cc102770 2024-02-20T07:50:55.122 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d205a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d205a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57114/0 (socket says 172.21.15.55:57114) 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.029+0000 7f58d205a700 1 -- 172.21.15.55:0/2220703308 learned_addr learned my addr 172.21.15.55:0/2220703308 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.030+0000 7f58d205a700 1 -- 172.21.15.55:0/2220703308 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc103200 con 0x7f58cc102770 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.030+0000 7f58d205a700 1 --2- 172.21.15.55:0/2220703308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58c80098e0 tx=0x7f58c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b613c38a3ec92d7e server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.030+0000 7f58d1058700 1 -- 172.21.15.55:0/2220703308 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58c800b860 con 0x7f58cc102770 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.030+0000 7f58d1058700 1 -- 172.21.15.55:0/2220703308 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f58c800be50 con 0x7f58cc102770 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.030+0000 7f58d1058700 1 -- 172.21.15.55:0/2220703308 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58c8010380 con 0x7f58cc102770 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 -- 172.21.15.55:0/2220703308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 msgr2=0x7f58cc102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 --2- 172.21.15.55:0/2220703308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58c80098e0 tx=0x7f58c8009250 comp rx=0 tx=0).stop 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 -- 172.21.15.55:0/2220703308 shutdown_connections 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 --2- 172.21.15.55:0/2220703308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 -- 172.21.15.55:0/2220703308 >> 172.21.15.55:0/2220703308 conn(0x7f58cc0fded0 msgr2=0x7f58cc100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 -- 172.21.15.55:0/2220703308 shutdown_connections 2024-02-20T07:50:55.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.031+0000 7f58d42be700 1 -- 172.21.15.55:0/2220703308 wait complete. 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d42be700 1 Processor -- start 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d42be700 1 -- start start 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d42be700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d42be700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc077190 con 0x7f58cc102770 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d205a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d205a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57128/0 (socket says 172.21.15.55:57128) 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d205a700 1 -- 172.21.15.55:0/2639810532 learned_addr learned my addr 172.21.15.55:0/2639810532 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d205a700 1 -- 172.21.15.55:0/2639810532 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c80094f0 con 0x7f58cc102770 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58d205a700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f58c800b440 tx=0x7f58c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.032+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58c80107c0 con 0x7f58cc102770 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.033+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f58c8010db0 con 0x7f58cc102770 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.033+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc077390 con 0x7f58cc102770 2024-02-20T07:50:55.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.033+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58c801ada0 con 0x7f58cc102770 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.033+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc077820 con 0x7f58cc102770 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.033+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f58c802e440 con 0x7f58cc102770 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.034+0000 7f58c2ffd700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.034+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 --> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58b80393a0 con 0x7f58b80367e0 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.034+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f58c804ac30 con 0x7f58cc102770 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.034+0000 7f58d1859700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 msgr2=0x7f58b8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.034+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.235+0000 7f58d1859700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 msgr2=0x7f58b8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.235+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.636+0000 7f58d1859700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 msgr2=0x7f58b8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:49.636+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:50.437+0000 7f58d1859700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 msgr2=0x7f58b8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:50.437+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-20T07:50:55.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:52.038+0000 7f58d1859700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 msgr2=0x7f58b8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/117641079 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:52.038+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:53.625+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f58c802ecf0 con 0x7f58cc102770 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:53.625+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 msgr2=0x7f58b8038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:53.625+0000 7f58c2ffd700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.621+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f58c802e970 con 0x7f58cc102770 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.621+0000 7f58c2ffd700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f58b8039fc0 0x7f58b803c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.621+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58b80393a0 con 0x7f58b8039fc0 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.622+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f58b8039fc0 0x7f58b803c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.623+0000 7f58d1859700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f58b8039fc0 0x7f58b803c3a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0039d0 tx=0x7f58bc009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.623+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f58b80393a0 con 0x7f58b8039fc0 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.629+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f58cc05f5c0 con 0x7f58b8039fc0 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.629+0000 7f58c2ffd700 1 -- 172.21.15.55:0/2639810532 <== mgr.14118 v2:172.21.15.55:6800/4027903862 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f58cc05f5c0 con 0x7f58b8039fc0 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f58b8039fc0 msgr2=0x7f58b803c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f58b8039fc0 0x7f58b803c3a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0039d0 tx=0x7f58bc009350 comp rx=0 tx=0).stop 2024-02-20T07:50:55.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 msgr2=0x7f58cc078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f58c800b440 tx=0x7f58c8009250 comp rx=0 tx=0).stop 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 shutdown_connections 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f58b8039fc0 0x7f58b803c3a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58cc102770 0x7f58cc078d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 --2- 172.21.15.55:0/2639810532 >> [v2:172.21.15.55:6800/117641079,v1:172.21.15.55:6801/117641079] conn(0x7f58b80367e0 0x7f58b8038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 >> 172.21.15.55:0/2639810532 conn(0x7f58cc0fded0 msgr2=0x7f58cc0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 shutdown_connections 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:54.630+0000 7f58d42be700 1 -- 172.21.15.55:0/2639810532 wait complete. 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:mgr epoch 5 is available 2024-02-20T07:50:55.127 INFO:teuthology.orchestra.run.smithi055.stdout:Setting orchestrator backend to cephadm... 2024-02-20T07:50:55.808 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:55 smithi055 ceph-mon[29254]: mgrmap e7: smithi055.stjper(active, since 1.00523s) 2024-02-20T07:50:55.808 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:55 smithi055 ceph-mon[29254]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T07:50:55.808 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:55 smithi055 ceph-mon[29254]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T07:50:55.808 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:55 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:50:55.808 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:55 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:56.304 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2abd84700 1 Processor -- start 2024-02-20T07:50:56.304 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2abd84700 1 -- start start 2024-02-20T07:50:56.304 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2abd84700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a40771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:56.304 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2abd84700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40790d0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2a9b20700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a40771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2a9b20700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a40771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47010/0 (socket says 172.21.15.55:47010) 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.573+0000 7ff2a9b20700 1 -- 172.21.15.55:0/1801548145 learned_addr learned my addr 172.21.15.55:0/1801548145 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2a9b20700 1 -- 172.21.15.55:0/1801548145 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4077710 con 0x7ff2a4078cc0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2a9b20700 1 --2- 172.21.15.55:0/1801548145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a40771d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00098e0 tx=0x7ff2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb43cca83356e405 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2a8b1e700 1 -- 172.21.15.55:0/1801548145 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a000b860 con 0x7ff2a4078cc0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2a8b1e700 1 -- 172.21.15.55:0/1801548145 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff2a000be50 con 0x7ff2a4078cc0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2a8b1e700 1 -- 172.21.15.55:0/1801548145 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a0010380 con 0x7ff2a4078cc0 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2abd84700 1 -- 172.21.15.55:0/1801548145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 msgr2=0x7ff2a40771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.574+0000 7ff2abd84700 1 --2- 172.21.15.55:0/1801548145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a40771d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00098e0 tx=0x7ff2a0009250 comp rx=0 tx=0).stop 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 -- 172.21.15.55:0/1801548145 shutdown_connections 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 --2- 172.21.15.55:0/1801548145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a40771d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:56.305 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 -- 172.21.15.55:0/1801548145 >> 172.21.15.55:0/1801548145 conn(0x7ff2a40fded0 msgr2=0x7ff2a4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:56.306 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 -- 172.21.15.55:0/1801548145 shutdown_connections 2024-02-20T07:50:56.306 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 -- 172.21.15.55:0/1801548145 wait complete. 2024-02-20T07:50:56.306 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 Processor -- start 2024-02-20T07:50:56.306 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 -- start start 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a41137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2abd84700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4113d10 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2a9b20700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a41137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2a9b20700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a41137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47012/0 (socket says 172.21.15.55:47012) 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.575+0000 7ff2a9b20700 1 -- 172.21.15.55:0/613693864 learned_addr learned my addr 172.21.15.55:0/613693864 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff2a9b20700 1 -- 172.21.15.55:0/613693864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00094f0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff2a9b20700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a41137d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0006fd0 tx=0x7ff2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a00107c0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff2a0010db0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a4113f10 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a001ada0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.576+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a4114bb0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.577+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7ff2a0019410 con 0x7ff2a4078cc0 2024-02-20T07:50:56.307 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.577+0000 7ff29a7fc700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff2900366b0 0x7ff290038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.577+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a405f5c0 con 0x7ff2a4078cc0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.577+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff2a0049c50 con 0x7ff2a4078cc0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.577+0000 7ff2a931f700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff2900366b0 0x7ff290038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.577+0000 7ff2a931f700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff2900366b0 0x7ff290038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff294009990 tx=0x7ff294006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.582+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff2a0018e20 con 0x7ff2a4078cc0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.786+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2a4100240 con 0x7ff2900366b0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.792+0000 7ff29a7fc700 1 -- 172.21.15.55:0/613693864 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff2a4100240 con 0x7ff2900366b0 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.794+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff2900366b0 msgr2=0x7ff290038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.794+0000 7ff2abd84700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff2900366b0 0x7ff290038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff294009990 tx=0x7ff294006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.794+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 msgr2=0x7ff2a41137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.794+0000 7ff2abd84700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a41137d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0006fd0 tx=0x7ff2a0009250 comp rx=0 tx=0).stop 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.795+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 shutdown_connections 2024-02-20T07:50:56.308 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.795+0000 7ff2abd84700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff2900366b0 0x7ff290038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:56.309 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.795+0000 7ff2abd84700 1 --2- 172.21.15.55:0/613693864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2a4078cc0 0x7ff2a41137d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:56.309 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.795+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 >> 172.21.15.55:0/613693864 conn(0x7ff2a40fded0 msgr2=0x7ff2a40ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:56.309 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.795+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 shutdown_connections 2024-02-20T07:50:56.309 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:55.795+0000 7ff2abd84700 1 -- 172.21.15.55:0/613693864 wait complete. 2024-02-20T07:50:56.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: [20/Feb/2024:07:50:55] ENGINE Bus STARTING 2024-02-20T07:50:56.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: [20/Feb/2024:07:50:55] ENGINE Serving on https://172.21.15.55:7150 2024-02-20T07:50:56.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: [20/Feb/2024:07:50:55] ENGINE Bus STARTED 2024-02-20T07:50:56.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:50:56.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:50:56.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:56.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:56 smithi055 ceph-mon[29254]: mgrmap e8: smithi055.stjper(active, since 3s) 2024-02-20T07:50:57.461 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.694+0000 7f4f4b7ec700 1 Processor -- start 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.694+0000 7f4f4b7ec700 1 -- start start 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.694+0000 7f4f4b7ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f440760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.694+0000 7f4f4b7ec700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44077fc0 con 0x7f4f44077bb0 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f49588700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f440760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f49588700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f440760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47014/0 (socket says 172.21.15.55:47014) 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f49588700 1 -- 172.21.15.55:0/267111203 learned_addr learned my addr 172.21.15.55:0/267111203 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f49588700 1 -- 172.21.15.55:0/267111203 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44076600 con 0x7f4f44077bb0 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f49588700 1 --2- 172.21.15.55:0/267111203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f440760c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f340098e0 tx=0x7f4f34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d158bc02ed3b05 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f3bfff700 1 -- 172.21.15.55:0/267111203 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3400b860 con 0x7f4f44077bb0 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f3bfff700 1 -- 172.21.15.55:0/267111203 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4f3400be50 con 0x7f4f44077bb0 2024-02-20T07:50:57.462 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.695+0000 7f4f3bfff700 1 -- 172.21.15.55:0/267111203 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f34010380 con 0x7f4f44077bb0 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/267111203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 msgr2=0x7f4f440760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 --2- 172.21.15.55:0/267111203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f440760c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f340098e0 tx=0x7f4f34009250 comp rx=0 tx=0).stop 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/267111203 shutdown_connections 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 --2- 172.21.15.55:0/267111203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f440760c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/267111203 >> 172.21.15.55:0/267111203 conn(0x7f4f440fded0 msgr2=0x7f4f44100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/267111203 shutdown_connections 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/267111203 wait complete. 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.696+0000 7f4f4b7ec700 1 Processor -- start 2024-02-20T07:50:57.463 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f4b7ec700 1 -- start start 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f4b7ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f44113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f4b7ec700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44113fc0 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f49588700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f44113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f49588700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f44113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47030/0 (socket says 172.21.15.55:47030) 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f49588700 1 -- 172.21.15.55:0/56357314 learned_addr learned my addr 172.21.15.55:0/56357314 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f49588700 1 -- 172.21.15.55:0/56357314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340094f0 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f49588700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f44113a80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f34006fd0 tx=0x7f4f34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f340107c0 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4f34010db0 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3401adb0 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.697+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f441141c0 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.698+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f44114e60 con 0x7f4f44077bb0 2024-02-20T07:50:57.464 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.698+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4f3401a650 con 0x7f4f44077bb0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.698+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4405f5c0 con 0x7f4f44077bb0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.698+0000 7f4f39ffb700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f4f300367e0 0x7f4f30038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.699+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4f3401e070 con 0x7f4f44077bb0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.699+0000 7f4f48d87700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f4f300367e0 0x7f4f30038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.699+0000 7f4f48d87700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f4f300367e0 0x7f4f30038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009990 tx=0x7f4f40006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.703+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4f34018b40 con 0x7f4f44077bb0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.896+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f44108bc0 con 0x7f4f300367e0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.897+0000 7f4f39ffb700 1 -- 172.21.15.55:0/56357314 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4f44108bc0 con 0x7f4f300367e0 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.898+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f4f300367e0 msgr2=0x7f4f30038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f4f300367e0 0x7f4f30038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009990 tx=0x7f4f40006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 msgr2=0x7f4f44113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f44113a80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f34006fd0 tx=0x7f4f34009250 comp rx=0 tx=0).stop 2024-02-20T07:50:57.465 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 shutdown_connections 2024-02-20T07:50:57.466 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f4f300367e0 0x7f4f30038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:57.466 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 --2- 172.21.15.55:0/56357314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f44077bb0 0x7f4f44113a80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:57.466 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 >> 172.21.15.55:0/56357314 conn(0x7f4f440fded0 msgr2=0x7f4f44100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:57.466 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 shutdown_connections 2024-02-20T07:50:57.466 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:56.899+0000 7f4f4b7ec700 1 -- 172.21.15.55:0/56357314 wait complete. 2024-02-20T07:50:57.466 INFO:teuthology.orchestra.run.smithi055.stdout:Generating ssh key... 2024-02-20T07:50:58.064 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:57 smithi055 ceph-mon[29254]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f0565a90700 1 Processor -- start 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f0565a90700 1 -- start start 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f0565a90700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f0565a90700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0560101fc0 con 0x7f0560101670 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f055f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f055f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47036/0 (socket says 172.21.15.55:47036) 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.858+0000 7f055f7fe700 1 -- 172.21.15.55:0/942637027 learned_addr learned my addr 172.21.15.55:0/942637027 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.859+0000 7f055f7fe700 1 -- 172.21.15.55:0/942637027 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0560102100 con 0x7f0560101670 2024-02-20T07:50:59.026 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.859+0000 7f055f7fe700 1 --2- 172.21.15.55:0/942637027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560101a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f05500098e0 tx=0x7f0550009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77df45d7e7497ba4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.859+0000 7f055e7fc700 1 -- 172.21.15.55:0/942637027 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f055000b860 con 0x7f0560101670 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.859+0000 7f055e7fc700 1 -- 172.21.15.55:0/942637027 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f055000be50 con 0x7f0560101670 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.859+0000 7f055e7fc700 1 -- 172.21.15.55:0/942637027 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0550010380 con 0x7f0560101670 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 -- 172.21.15.55:0/942637027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 msgr2=0x7f0560101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 --2- 172.21.15.55:0/942637027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560101a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f05500098e0 tx=0x7f0550009250 comp rx=0 tx=0).stop 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 -- 172.21.15.55:0/942637027 shutdown_connections 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 --2- 172.21.15.55:0/942637027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560101a80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 -- 172.21.15.55:0/942637027 >> 172.21.15.55:0/942637027 conn(0x7f05600fcdd0 msgr2=0x7f05600ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 -- 172.21.15.55:0/942637027 shutdown_connections 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 -- 172.21.15.55:0/942637027 wait complete. 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 Processor -- start 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 -- start start 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.860+0000 7f0565a90700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:59.027 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f0565a90700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0560111a80 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f055f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f055f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47038/0 (socket says 172.21.15.55:47038) 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f055f7fe700 1 -- 172.21.15.55:0/2073763336 learned_addr learned my addr 172.21.15.55:0/2073763336 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f055f7fe700 1 -- 172.21.15.55:0/2073763336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500094f0 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f055f7fe700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560111540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f055000b440 tx=0x7f0550009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0550010760 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0550010d50 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0560111c80 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f055001ae20 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.861+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0560112920 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.862+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f05500108c0 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.862+0000 7f0547fff700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f05480367e0 0x7f0548038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.862+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056005f5c0 con 0x7f0560101670 2024-02-20T07:50:59.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.862+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f055004a500 con 0x7f0560101670 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.862+0000 7f055effd700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f05480367e0 0x7f0548038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.863+0000 7f055effd700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f05480367e0 0x7f0548038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0554009990 tx=0x7f0554006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:57.867+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0550018b40 con 0x7f0560101670 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.062+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f05600fef30 con 0x7f05480367e0 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.444+0000 7f0547fff700 1 -- 172.21.15.55:0/2073763336 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f05600fef30 con 0x7f05480367e0 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.446+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f05480367e0 msgr2=0x7f0548038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.446+0000 7f0565a90700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f05480367e0 0x7f0548038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0554009990 tx=0x7f0554006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.446+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 msgr2=0x7f0560111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.446+0000 7f0565a90700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560111540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f055000b440 tx=0x7f0550009250 comp rx=0 tx=0).stop 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.446+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 shutdown_connections 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.446+0000 7f0565a90700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f05480367e0 0x7f0548038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.447+0000 7f0565a90700 1 --2- 172.21.15.55:0/2073763336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0560101670 0x7f0560111540 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.447+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 >> 172.21.15.55:0/2073763336 conn(0x7f05600fcdd0 msgr2=0x7f0560076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.447+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 shutdown_connections 2024-02-20T07:50:59.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:58.447+0000 7f0565a90700 1 -- 172.21.15.55:0/2073763336 wait complete. 2024-02-20T07:50:59.657 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:59 smithi055 ceph-mon[29254]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:50:59.657 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:59 smithi055 ceph-mon[29254]: Generating ssh key... 2024-02-20T07:50:59.657 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:59 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:50:59.657 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:50:59 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:00.229 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuJvGIPV7FFdFaQZp6plHTpj3SLxFP3boKwhAWCOra8LCoNKAwVyNXXsAjZ3vjd1tmsQMlhZI5Jq1xHarKktDD6IeBylOeUPxegISaqjIffeZZ3ABAEJdx6zbxJ+DK4u0rCTYhX+v7n8q+wWiFE2768tgpB1Qu/JMZjPt8iVeqRENoXiLrheR1ZVXReUstgk7qoA3JofBmSPAzrmS4tSMEBo3u/zwqQn68gUrMk/mTxUUmidxqZMidI1TRAoQvWpstRp4vpKQFv6PaqXMpqvMWXLWjZulRyW00yVXNo7IZhoLQxGcz8NkdiYnYDzf/sSobbOWjVWnmmk5uYcDX8rTuTXfRN19R1JxTmYXZPY1Vv6L90J1881Bkk9oWJn/3Eycw7uTii5QrDPmBGAsZJew34GIp/jySstE5eRQ4w0RHZiRxSR/fgI6/1e66S2bmXxK6X5RiV9EKdJM6uk0FwNHtUODFca/vrqWKQay6bJQ0S5SGsvgiell81Fz0pzWRRoc= ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:51:00.234 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7fb001283700 1 Processor -- start 2024-02-20T07:51:00.234 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7fb001283700 1 -- start start 2024-02-20T07:51:00.234 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7fb001283700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:00.234 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7fb001283700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1058a0 con 0x7faffc0714a0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7faffaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7faffaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47050/0 (socket says 172.21.15.55:47050) 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.452+0000 7faffaffd700 1 -- 172.21.15.55:0/360705009 learned_addr learned my addr 172.21.15.55:0/360705009 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7faffaffd700 1 -- 172.21.15.55:0/360705009 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc1059e0 con 0x7faffc0714a0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7faffaffd700 1 --2- 172.21.15.55:0/360705009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc105360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fafec0098e0 tx=0x7fafec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3b0f1d7fd0003 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7faff9ffb700 1 -- 172.21.15.55:0/360705009 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec00b860 con 0x7faffc0714a0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7faff9ffb700 1 -- 172.21.15.55:0/360705009 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fafec00be50 con 0x7faffc0714a0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7faff9ffb700 1 -- 172.21.15.55:0/360705009 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec010380 con 0x7faffc0714a0 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7fb001283700 1 -- 172.21.15.55:0/360705009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 msgr2=0x7faffc105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.453+0000 7fb001283700 1 --2- 172.21.15.55:0/360705009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc105360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fafec0098e0 tx=0x7fafec009250 comp rx=0 tx=0).stop 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 -- 172.21.15.55:0/360705009 shutdown_connections 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 --2- 172.21.15.55:0/360705009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc105360 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 -- 172.21.15.55:0/360705009 >> 172.21.15.55:0/360705009 conn(0x7faffc0fcdd0 msgr2=0x7faffc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 -- 172.21.15.55:0/360705009 shutdown_connections 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 -- 172.21.15.55:0/360705009 wait complete. 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 Processor -- start 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.454+0000 7fb001283700 1 -- start start 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fb001283700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fb001283700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc101260 con 0x7faffc0714a0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7faffaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7faffaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc102de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47054/0 (socket says 172.21.15.55:47054) 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7faffaffd700 1 -- 172.21.15.55:0/2672761594 learned_addr learned my addr 172.21.15.55:0/2672761594 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7faffaffd700 1 -- 172.21.15.55:0/2672761594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0094f0 con 0x7faffc0714a0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7faffaffd700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc102de0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fafec006b00 tx=0x7fafec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec0107c0 con 0x7faffc0714a0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fafec010db0 con 0x7faffc0714a0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc101460 con 0x7faffc0714a0 2024-02-20T07:51:00.236 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec01acf0 con 0x7faffc0714a0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.455+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc1018f0 con 0x7faffc0714a0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.456+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fafec0193c0 con 0x7faffc0714a0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.456+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc10b640 con 0x7faffc0714a0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.456+0000 7fafe37fe700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fafe4036790 0x7fafe4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.456+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fafec04a780 con 0x7faffc0714a0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.457+0000 7faffa7fc700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fafe4036790 0x7fafe4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.457+0000 7faffa7fc700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fafe4036790 0x7fafe4038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faff0009990 tx=0x7faff0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.461+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fafec018d60 con 0x7faffc0714a0 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.655+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7faffc02b410 con 0x7fafe4036790 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.655+0000 7fafe37fe700 1 -- 172.21.15.55:0/2672761594 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7faffc02b410 con 0x7fafe4036790 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fafe4036790 msgr2=0x7fafe4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fafe4036790 0x7fafe4038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faff0009990 tx=0x7faff0006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:00.237 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 msgr2=0x7faffc102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc102de0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fafec006b00 tx=0x7fafec009250 comp rx=0 tx=0).stop 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 shutdown_connections 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fafe4036790 0x7fafe4038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 --2- 172.21.15.55:0/2672761594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faffc0714a0 0x7faffc102de0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 >> 172.21.15.55:0/2672761594 conn(0x7faffc0fcdd0 msgr2=0x7faffc0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 shutdown_connections 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:50:59.657+0000 7fb001283700 1 -- 172.21.15.55:0/2672761594 wait complete. 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:Adding key to root@localhost authorized_keys... 2024-02-20T07:51:00.238 INFO:teuthology.orchestra.run.smithi055.stdout:Adding host smithi055... 2024-02-20T07:51:00.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:00 smithi055 ceph-mon[29254]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:01.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:01 smithi055 ceph-mon[29254]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:02.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:02 smithi055 ceph-mon[29254]: Deploying cephadm binary to smithi055 2024-02-20T07:51:03.477 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Added host 'smithi055' with addr '172.21.15.55' 2024-02-20T07:51:03.477 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f7632a4b700 1 Processor -- start 2024-02-20T07:51:03.477 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f7632a4b700 1 -- start start 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f7632a4b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f7632a4b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c1030c0 con 0x7f762c102770 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f762bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f762bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47064/0 (socket says 172.21.15.55:47064) 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.651+0000 7f762bfff700 1 -- 172.21.15.55:0/4021231783 learned_addr learned my addr 172.21.15.55:0/4021231783 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.652+0000 7f762bfff700 1 -- 172.21.15.55:0/4021231783 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c103200 con 0x7f762c102770 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.652+0000 7f762bfff700 1 --2- 172.21.15.55:0/4021231783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c102b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f761c0098e0 tx=0x7f761c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5365f6fb490ad72 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.652+0000 7f762b7fe700 1 -- 172.21.15.55:0/4021231783 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f761c00b860 con 0x7f762c102770 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.652+0000 7f762b7fe700 1 -- 172.21.15.55:0/4021231783 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f761c00be50 con 0x7f762c102770 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.652+0000 7f762b7fe700 1 -- 172.21.15.55:0/4021231783 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f761c010380 con 0x7f762c102770 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 -- 172.21.15.55:0/4021231783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 msgr2=0x7f762c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 --2- 172.21.15.55:0/4021231783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c102b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f761c0098e0 tx=0x7f761c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:03.478 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 -- 172.21.15.55:0/4021231783 shutdown_connections 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 --2- 172.21.15.55:0/4021231783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c102b80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 -- 172.21.15.55:0/4021231783 >> 172.21.15.55:0/4021231783 conn(0x7f762c0fded0 msgr2=0x7f762c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 -- 172.21.15.55:0/4021231783 shutdown_connections 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 -- 172.21.15.55:0/4021231783 wait complete. 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 Processor -- start 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.653+0000 7f7632a4b700 1 -- start start 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f7632a4b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f7632a4b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c111ad0 con 0x7f762c102770 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f762bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f762bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47070/0 (socket says 172.21.15.55:47070) 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f762bfff700 1 -- 172.21.15.55:0/4062722514 learned_addr learned my addr 172.21.15.55:0/4062722514 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f762bfff700 1 -- 172.21.15.55:0/4062722514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0094f0 con 0x7f762c102770 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f762bfff700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c111590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f761c00b440 tx=0x7f761c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:03.479 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f761c010760 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f761c010d50 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c111cd0 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f761c01ae20 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.654+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c112970 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.655+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f761c0108c0 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.655+0000 7f76297fa700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f76180367e0 0x7f7618038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.655+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f761c04a570 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.655+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c05f5c0 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.656+0000 7f7623fff700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f76180367e0 0x7f7618038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.656+0000 7f7623fff700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f76180367e0 0x7f7618038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7614009990 tx=0x7f7614006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.660+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f761c02e440 con 0x7f762c102770 2024-02-20T07:51:03.480 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:00.855+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}) v1 -- 0x7f762c1001b0 con 0x7f76180367e0 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.894+0000 7f76297fa700 1 -- 172.21.15.55:0/4062722514 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f762c1001b0 con 0x7f76180367e0 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f76180367e0 msgr2=0x7f7618038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f76180367e0 0x7f7618038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7614009990 tx=0x7f7614006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 msgr2=0x7f762c111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c111590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f761c00b440 tx=0x7f761c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 shutdown_connections 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f76180367e0 0x7f7618038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 --2- 172.21.15.55:0/4062722514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c102770 0x7f762c111590 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 >> 172.21.15.55:0/4062722514 conn(0x7f762c0fded0 msgr2=0x7f762c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 shutdown_connections 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:02.897+0000 7f7632a4b700 1 -- 172.21.15.55:0/4062722514 wait complete. 2024-02-20T07:51:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying mon service with default placement... 2024-02-20T07:51:04.100 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:03 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:04.100 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:03 smithi055 ceph-mon[29254]: Added host smithi055 2024-02-20T07:51:04.100 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:03 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:04.854 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-20T07:51:04.854 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae4704e700 1 Processor -- start 2024-02-20T07:51:04.854 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae4704e700 1 -- start start 2024-02-20T07:51:04.854 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae4704e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae4704e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae401030c0 con 0x7fae40102770 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae44dea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae44dea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49866/0 (socket says 172.21.15.55:49866) 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.864+0000 7fae44dea700 1 -- 172.21.15.55:0/3767075377 learned_addr learned my addr 172.21.15.55:0/3767075377 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.865+0000 7fae44dea700 1 -- 172.21.15.55:0/3767075377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae40103200 con 0x7fae40102770 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.865+0000 7fae44dea700 1 --2- 172.21.15.55:0/3767075377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40102b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fae340098e0 tx=0x7fae34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ab29fe17f97527 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.865+0000 7fae3f7fe700 1 -- 172.21.15.55:0/3767075377 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae3400b860 con 0x7fae40102770 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.865+0000 7fae3f7fe700 1 -- 172.21.15.55:0/3767075377 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fae3400be50 con 0x7fae40102770 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.865+0000 7fae3f7fe700 1 -- 172.21.15.55:0/3767075377 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae34010380 con 0x7fae40102770 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 -- 172.21.15.55:0/3767075377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 msgr2=0x7fae40102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 --2- 172.21.15.55:0/3767075377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40102b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fae340098e0 tx=0x7fae34009250 comp rx=0 tx=0).stop 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 -- 172.21.15.55:0/3767075377 shutdown_connections 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 --2- 172.21.15.55:0/3767075377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40102b80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:04.855 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 -- 172.21.15.55:0/3767075377 >> 172.21.15.55:0/3767075377 conn(0x7fae400fded0 msgr2=0x7fae40100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 -- 172.21.15.55:0/3767075377 shutdown_connections 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 -- 172.21.15.55:0/3767075377 wait complete. 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.866+0000 7fae4704e700 1 Processor -- start 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae4704e700 1 -- start start 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae4704e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae4704e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40111a40 con 0x7fae40102770 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae44dea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae44dea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49876/0 (socket says 172.21.15.55:49876) 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae44dea700 1 -- 172.21.15.55:0/3611689563 learned_addr learned my addr 172.21.15.55:0/3611689563 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae44dea700 1 -- 172.21.15.55:0/3611689563 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae340094f0 con 0x7fae40102770 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.867+0000 7fae44dea700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40111500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fae3400b440 tx=0x7fae34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.868+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae340107c0 con 0x7fae40102770 2024-02-20T07:51:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.868+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fae34010db0 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.868+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae40111c40 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.868+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae3401ada0 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.868+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae401128e0 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.869+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4005f5c0 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.869+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fae3402e440 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.869+0000 7fae3d7fa700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fae300367e0 0x7fae30038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.869+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fae3404ac90 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.869+0000 7fae3ffff700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fae300367e0 0x7fae30038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.869+0000 7fae3ffff700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fae300367e0 0x7fae30038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009990 tx=0x7fae2c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:03.874+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fae34024440 con 0x7fae40102770 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.098+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fae400ff380 con 0x7fae300367e0 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.103+0000 7fae3d7fa700 1 -- 172.21.15.55:0/3611689563 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fae400ff380 con 0x7fae300367e0 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.104+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fae300367e0 msgr2=0x7fae30038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.104+0000 7fae4704e700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fae300367e0 0x7fae30038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009990 tx=0x7fae2c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.104+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 msgr2=0x7fae40111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.104+0000 7fae4704e700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40111500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fae3400b440 tx=0x7fae34009250 comp rx=0 tx=0).stop 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.105+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 shutdown_connections 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.105+0000 7fae4704e700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fae300367e0 0x7fae30038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.105+0000 7fae4704e700 1 --2- 172.21.15.55:0/3611689563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae40102770 0x7fae40111500 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.105+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 >> 172.21.15.55:0/3611689563 conn(0x7fae400fded0 msgr2=0x7fae400fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.105+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 shutdown_connections 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:04.105+0000 7fae4704e700 1 -- 172.21.15.55:0/3611689563 wait complete. 2024-02-20T07:51:04.858 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying mgr service with default placement... 2024-02-20T07:51:05.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:05 smithi055 ceph-mon[29254]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:05.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:05 smithi055 ceph-mon[29254]: Saving service mon spec with placement count:5 2024-02-20T07:51:05.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:05 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1ee17e5700 1 Processor -- start 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1ee17e5700 1 -- start start 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1ee17e5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1ee17e5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc1058a0 con 0x7f1edc076c40 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1edaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1edaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49882/0 (socket says 172.21.15.55:49882) 2024-02-20T07:51:06.081 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1edaffd700 1 -- 172.21.15.55:0/189233246 learned_addr learned my addr 172.21.15.55:0/189233246 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.293+0000 7f1edaffd700 1 -- 172.21.15.55:0/189233246 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc1059e0 con 0x7f1edc076c40 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1edaffd700 1 --2- 172.21.15.55:0/189233246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc105360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0098e0 tx=0x7f1ecc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba011f1a575cd02 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ed9ffb700 1 -- 172.21.15.55:0/189233246 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ecc00b860 con 0x7f1edc076c40 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ed9ffb700 1 -- 172.21.15.55:0/189233246 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1ecc00be50 con 0x7f1edc076c40 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ed9ffb700 1 -- 172.21.15.55:0/189233246 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ecc010380 con 0x7f1edc076c40 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ee17e5700 1 -- 172.21.15.55:0/189233246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 msgr2=0x7f1edc105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ee17e5700 1 --2- 172.21.15.55:0/189233246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc105360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0098e0 tx=0x7f1ecc009250 comp rx=0 tx=0).stop 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ee17e5700 1 -- 172.21.15.55:0/189233246 shutdown_connections 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ee17e5700 1 --2- 172.21.15.55:0/189233246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc105360 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ee17e5700 1 -- 172.21.15.55:0/189233246 >> 172.21.15.55:0/189233246 conn(0x7f1edc0fcdd0 msgr2=0x7f1edc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.294+0000 7f1ee17e5700 1 -- 172.21.15.55:0/189233246 shutdown_connections 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.295+0000 7f1ee17e5700 1 -- 172.21.15.55:0/189233246 wait complete. 2024-02-20T07:51:06.082 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.295+0000 7f1ee17e5700 1 Processor -- start 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.295+0000 7f1ee17e5700 1 -- start start 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.295+0000 7f1ee17e5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.295+0000 7f1ee17e5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc111d60 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1edaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1edaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49884/0 (socket says 172.21.15.55:49884) 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1edaffd700 1 -- 172.21.15.55:0/553597155 learned_addr learned my addr 172.21.15.55:0/553597155 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1edaffd700 1 -- 172.21.15.55:0/553597155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc0094f0 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1edaffd700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc111820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc006d40 tx=0x7f1ecc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ecc010760 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc111f60 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1ecc010d50 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ecc024d20 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.296+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc112c00 con 0x7f1edc076c40 2024-02-20T07:51:06.083 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.297+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1ecc02e440 con 0x7f1edc076c40 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.297+0000 7f1ec37fe700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f1ec40367e0 0x7f1ec4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.297+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1ecc04a890 con 0x7f1edc076c40 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.297+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc05f5c0 con 0x7f1edc076c40 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.297+0000 7f1eda7fc700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f1ec40367e0 0x7f1ec4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.298+0000 7f1eda7fc700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f1ec40367e0 0x7f1ec4038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009990 tx=0x7f1ed0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.302+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1ecc018940 con 0x7f1edc076c40 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.507+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1edc1067f0 con 0x7f1ec40367e0 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.514+0000 7f1ec37fe700 1 -- 172.21.15.55:0/553597155 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1edc1067f0 con 0x7f1ec40367e0 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f1ec40367e0 msgr2=0x7f1ec4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f1ec40367e0 0x7f1ec4038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009990 tx=0x7f1ed0006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 msgr2=0x7f1edc111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc111820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc006d40 tx=0x7f1ecc009250 comp rx=0 tx=0).stop 2024-02-20T07:51:06.084 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 shutdown_connections 2024-02-20T07:51:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f1ec40367e0 0x7f1ec4038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 --2- 172.21.15.55:0/553597155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1edc076c40 0x7f1edc111820 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.516+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 >> 172.21.15.55:0/553597155 conn(0x7f1edc0fcdd0 msgr2=0x7f1edc0ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.517+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 shutdown_connections 2024-02-20T07:51:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:05.517+0000 7f1ee17e5700 1 -- 172.21.15.55:0/553597155 wait complete. 2024-02-20T07:51:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying crash service with default placement... 2024-02-20T07:51:06.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:06 smithi055 ceph-mon[29254]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:06.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:06 smithi055 ceph-mon[29254]: Saving service mgr spec with placement count:2 2024-02-20T07:51:06.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:06 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.508+0000 7fb804d42700 1 Processor -- start 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.508+0000 7fb804d42700 1 -- start start 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.508+0000 7fb804d42700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.508+0000 7fb804d42700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800103590 con 0x7fb800102c40 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49900/0 (socket says 172.21.15.55:49900) 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fe59c700 1 -- 172.21.15.55:0/1311465020 learned_addr learned my addr 172.21.15.55:0/1311465020 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fe59c700 1 -- 172.21.15.55:0/1311465020 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8001036d0 con 0x7fb800102c40 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fe59c700 1 --2- 172.21.15.55:0/1311465020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800103050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009e50 tx=0x7fb7f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47dadf59c5c4381 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fd59a700 1 -- 172.21.15.55:0/1311465020 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f00092e0 con 0x7fb800102c40 2024-02-20T07:51:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fd59a700 1 -- 172.21.15.55:0/1311465020 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb7f0012430 con 0x7fb800102c40 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.509+0000 7fb7fd59a700 1 -- 172.21.15.55:0/1311465020 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f00075c0 con 0x7fb800102c40 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 -- 172.21.15.55:0/1311465020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 msgr2=0x7fb800103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 --2- 172.21.15.55:0/1311465020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800103050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009e50 tx=0x7fb7f0009840 comp rx=0 tx=0).stop 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 -- 172.21.15.55:0/1311465020 shutdown_connections 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 --2- 172.21.15.55:0/1311465020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800103050 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 -- 172.21.15.55:0/1311465020 >> 172.21.15.55:0/1311465020 conn(0x7fb8000fc580 msgr2=0x7fb8000fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 -- 172.21.15.55:0/1311465020 shutdown_connections 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 -- 172.21.15.55:0/1311465020 wait complete. 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.510+0000 7fb804d42700 1 Processor -- start 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb804d42700 1 -- start start 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb804d42700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800107a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb804d42700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800105ee0 con 0x7fb800102c40 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7fe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800107a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:07.927 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7fe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800107a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49904/0 (socket says 172.21.15.55:49904) 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7fe59c700 1 -- 172.21.15.55:0/3115963911 learned_addr learned my addr 172.21.15.55:0/3115963911 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7fe59c700 1 -- 172.21.15.55:0/3115963911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f0009ae0 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7fe59c700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800107a60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00127d0 tx=0x7fb7f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f000bc30 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb7f000bd90 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8001060e0 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.511+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f001c9d0 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.512+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8001065d0 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.513+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80010b740 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.513+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb7f0019070 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.513+0000 7fb7f6ffd700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb7ec036790 0x7fb7ec038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.513+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7f0049e90 con 0x7fb800102c40 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.513+0000 7fb7fdd9b700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb7ec036790 0x7fb7ec038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:07.928 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.513+0000 7fb7fdd9b700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb7ec036790 0x7fb7ec038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009990 tx=0x7fb7e8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.520+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb7f0007880 con 0x7fb800102c40 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.725+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb800102570 con 0x7fb7ec036790 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.730+0000 7fb7f6ffd700 1 -- 172.21.15.55:0/3115963911 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb800102570 con 0x7fb7ec036790 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb7ec036790 msgr2=0x7fb7ec038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb7ec036790 0x7fb7ec038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009990 tx=0x7fb7e8006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 msgr2=0x7fb800107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800107a60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00127d0 tx=0x7fb7f0009840 comp rx=0 tx=0).stop 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 shutdown_connections 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb7ec036790 0x7fb7ec038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 --2- 172.21.15.55:0/3115963911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb800102c40 0x7fb800107a60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 >> 172.21.15.55:0/3115963911 conn(0x7fb8000fc580 msgr2=0x7fb8000fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 shutdown_connections 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:06.732+0000 7fb804d42700 1 -- 172.21.15.55:0/3115963911 wait complete. 2024-02-20T07:51:07.929 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying prometheus service with default placement... 2024-02-20T07:51:08.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:07 smithi055 ceph-mon[29254]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:08.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:07 smithi055 ceph-mon[29254]: Saving service crash spec with placement * 2024-02-20T07:51:08.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:07 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:08.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:07 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:09.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:09 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:09.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:09 smithi055 ceph-mon[29254]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:09.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:09 smithi055 ceph-mon[29254]: Saving service prometheus spec with placement count:1 2024-02-20T07:51:09.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:09 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:09.706 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-20T07:51:09.706 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd577f25700 1 Processor -- start 2024-02-20T07:51:09.706 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd577f25700 1 -- start start 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd577f25700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd5701077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd577f25700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd570103190 con 0x7fd570076c00 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd575cc1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd5701077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd575cc1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd5701077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49906/0 (socket says 172.21.15.55:49906) 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.337+0000 7fd575cc1700 1 -- 172.21.15.55:0/2836219093 learned_addr learned my addr 172.21.15.55:0/2836219093 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.338+0000 7fd575cc1700 1 -- 172.21.15.55:0/2836219093 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570107ce0 con 0x7fd570076c00 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.338+0000 7fd575cc1700 1 --2- 172.21.15.55:0/2836219093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd5701077a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0098e0 tx=0x7fd56c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9921d8a695d3513 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.338+0000 7fd574cbf700 1 -- 172.21.15.55:0/2836219093 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56c00b860 con 0x7fd570076c00 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.338+0000 7fd574cbf700 1 -- 172.21.15.55:0/2836219093 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56c00be90 con 0x7fd570076c00 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.338+0000 7fd574cbf700 1 -- 172.21.15.55:0/2836219093 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56c0103c0 con 0x7fd570076c00 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 -- 172.21.15.55:0/2836219093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 msgr2=0x7fd5701077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 --2- 172.21.15.55:0/2836219093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd5701077a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0098e0 tx=0x7fd56c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 -- 172.21.15.55:0/2836219093 shutdown_connections 2024-02-20T07:51:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 --2- 172.21.15.55:0/2836219093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd5701077a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 -- 172.21.15.55:0/2836219093 >> 172.21.15.55:0/2836219093 conn(0x7fd5700fcdd0 msgr2=0x7fd5700ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 -- 172.21.15.55:0/2836219093 shutdown_connections 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.339+0000 7fd577f25700 1 -- 172.21.15.55:0/2836219093 wait complete. 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd577f25700 1 Processor -- start 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd577f25700 1 -- start start 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd577f25700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd570113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd577f25700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd570113ca0 con 0x7fd570076c00 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd575cc1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd570113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd575cc1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd570113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49912/0 (socket says 172.21.15.55:49912) 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd575cc1700 1 -- 172.21.15.55:0/27527831 learned_addr learned my addr 172.21.15.55:0/27527831 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.340+0000 7fd575cc1700 1 -- 172.21.15.55:0/27527831 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0094f0 con 0x7fd570076c00 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.341+0000 7fd575cc1700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd570113760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd56c003080 tx=0x7fd56c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.341+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56c011920 con 0x7fd570076c00 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.341+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56c010d60 con 0x7fd570076c00 2024-02-20T07:51:09.708 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.341+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd570113ea0 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.341+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56c0243f0 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.341+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570114b40 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.342+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd56c02d3f0 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.342+0000 7fd5667fc700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fd55c0367e0 0x7fd55c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.342+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd56c04a310 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.342+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57005f5c0 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.342+0000 7fd5754c0700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fd55c0367e0 0x7fd55c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.343+0000 7fd5754c0700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fd55c0367e0 0x7fd55c038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd560009990 tx=0x7fd560006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.347+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd56c02b8e0 con 0x7fd570076c00 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.577+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5700fe2e0 con 0x7fd55c0367e0 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.582+0000 7fd5667fc700 1 -- 172.21.15.55:0/27527831 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd5700fe2e0 con 0x7fd55c0367e0 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fd55c0367e0 msgr2=0x7fd55c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fd55c0367e0 0x7fd55c038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd560009990 tx=0x7fd560006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 msgr2=0x7fd570113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd570113760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd56c003080 tx=0x7fd56c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 shutdown_connections 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fd55c0367e0 0x7fd55c038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 --2- 172.21.15.55:0/27527831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd570076c00 0x7fd570113760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 >> 172.21.15.55:0/27527831 conn(0x7fd5700fcdd0 msgr2=0x7fd5700fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 shutdown_connections 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:08.584+0000 7fd577f25700 1 -- 172.21.15.55:0/27527831 wait complete. 2024-02-20T07:51:09.710 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying grafana service with default placement... 2024-02-20T07:51:10.914 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:10 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:10.914 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:10 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:10.921 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb10716d700 1 Processor -- start 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb10716d700 1 -- start start 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb10716d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb10716d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb100101fc0 con 0x7fb100101670 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb104f09700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb104f09700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49916/0 (socket says 172.21.15.55:49916) 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.146+0000 7fb104f09700 1 -- 172.21.15.55:0/10707567 learned_addr learned my addr 172.21.15.55:0/10707567 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb104f09700 1 -- 172.21.15.55:0/10707567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb100102100 con 0x7fb100101670 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb104f09700 1 --2- 172.21.15.55:0/10707567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100101a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40098e0 tx=0x7fb0f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8bbe8bcee932f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb0ff7fe700 1 -- 172.21.15.55:0/10707567 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f400b860 con 0x7fb100101670 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb0ff7fe700 1 -- 172.21.15.55:0/10707567 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0f400be90 con 0x7fb100101670 2024-02-20T07:51:10.922 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb0ff7fe700 1 -- 172.21.15.55:0/10707567 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f40103c0 con 0x7fb100101670 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb10716d700 1 -- 172.21.15.55:0/10707567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 msgr2=0x7fb100101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb10716d700 1 --2- 172.21.15.55:0/10707567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100101a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40098e0 tx=0x7fb0f4009250 comp rx=0 tx=0).stop 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb10716d700 1 -- 172.21.15.55:0/10707567 shutdown_connections 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.147+0000 7fb10716d700 1 --2- 172.21.15.55:0/10707567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100101a80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 -- 172.21.15.55:0/10707567 >> 172.21.15.55:0/10707567 conn(0x7fb1000fcdd0 msgr2=0x7fb1000ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 -- 172.21.15.55:0/10707567 shutdown_connections 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 -- 172.21.15.55:0/10707567 wait complete. 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 Processor -- start 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 -- start start 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100114520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb10716d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb100076c50 con 0x7fb100101670 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.148+0000 7fb104f09700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100114520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb104f09700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100114520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49928/0 (socket says 172.21.15.55:49928) 2024-02-20T07:51:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb104f09700 1 -- 172.21.15.55:0/2360209762 learned_addr learned my addr 172.21.15.55:0/2360209762 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb104f09700 1 -- 172.21.15.55:0/2360209762 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40094f0 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb104f09700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100114520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4006b00 tx=0x7fb0f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f40109a0 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0f401a430 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb100076e50 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f40245b0 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.149+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb100114ca0 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.150+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb0f401aaa0 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.150+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10010a590 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.150+0000 7fb0fd7fa700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb0f0036790 0x7fb0f0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.150+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb0f404a660 con 0x7fb100101670 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.150+0000 7fb0fffff700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb0f0036790 0x7fb0f0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:10.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.151+0000 7fb0fffff700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb0f0036790 0x7fb0f0038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009990 tx=0x7fb0ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.155+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb0f4018b40 con 0x7fb100101670 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.357+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fb10005e3d0 con 0x7fb0f0036790 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.361+0000 7fb0fd7fa700 1 -- 172.21.15.55:0/2360209762 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb10005e3d0 con 0x7fb0f0036790 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb0f0036790 msgr2=0x7fb0f0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb0f0036790 0x7fb0f0038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009990 tx=0x7fb0ec006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 msgr2=0x7fb100114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100114520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4006b00 tx=0x7fb0f4009250 comp rx=0 tx=0).stop 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 shutdown_connections 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb0f0036790 0x7fb0f0038c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 --2- 172.21.15.55:0/2360209762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb100101670 0x7fb100114520 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 >> 172.21.15.55:0/2360209762 conn(0x7fb1000fcdd0 msgr2=0x7fb1000fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.363+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 shutdown_connections 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:10.364+0000 7fb10716d700 1 -- 172.21.15.55:0/2360209762 wait complete. 2024-02-20T07:51:10.925 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying node-exporter service with default placement... 2024-02-20T07:51:12.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:11 smithi055 ceph-mon[29254]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:12.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:11 smithi055 ceph-mon[29254]: Saving service grafana spec with placement count:1 2024-02-20T07:51:12.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:11 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:12.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:11 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:12.123 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f9085f7d700 1 Processor -- start 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f9085f7d700 1 -- start start 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f9085f7d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90800771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f9085f7d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90800790f0 con 0x7f9080078ce0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f907f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90800771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f907f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90800771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49938/0 (socket says 172.21.15.55:49938) 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.340+0000 7f907f7fe700 1 -- 172.21.15.55:0/2241010533 learned_addr learned my addr 172.21.15.55:0/2241010533 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f907f7fe700 1 -- 172.21.15.55:0/2241010533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080077730 con 0x7f9080078ce0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f907f7fe700 1 --2- 172.21.15.55:0/2241010533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90800771f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90700098e0 tx=0x7f9070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583c6665e9dd2c97 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f907e7fc700 1 -- 172.21.15.55:0/2241010533 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f907000b860 con 0x7f9080078ce0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f907e7fc700 1 -- 172.21.15.55:0/2241010533 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f907000be90 con 0x7f9080078ce0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f907e7fc700 1 -- 172.21.15.55:0/2241010533 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90700103c0 con 0x7f9080078ce0 2024-02-20T07:51:12.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f9085f7d700 1 -- 172.21.15.55:0/2241010533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 msgr2=0x7f90800771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f9085f7d700 1 --2- 172.21.15.55:0/2241010533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90800771f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90700098e0 tx=0x7f9070009250 comp rx=0 tx=0).stop 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f9085f7d700 1 -- 172.21.15.55:0/2241010533 shutdown_connections 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f9085f7d700 1 --2- 172.21.15.55:0/2241010533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90800771f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.341+0000 7f9085f7d700 1 -- 172.21.15.55:0/2241010533 >> 172.21.15.55:0/2241010533 conn(0x7f90800fded0 msgr2=0x7f9080100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f9085f7d700 1 -- 172.21.15.55:0/2241010533 shutdown_connections 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f9085f7d700 1 -- 172.21.15.55:0/2241010533 wait complete. 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f9085f7d700 1 Processor -- start 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f9085f7d700 1 -- start start 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f9085f7d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90801137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f9085f7d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9080113d20 con 0x7f9080078ce0 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f907f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90801137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f907f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90801137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49940/0 (socket says 172.21.15.55:49940) 2024-02-20T07:51:12.125 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.342+0000 7f907f7fe700 1 -- 172.21.15.55:0/2687476221 learned_addr learned my addr 172.21.15.55:0/2687476221 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f907f7fe700 1 -- 172.21.15.55:0/2687476221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700094f0 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f907f7fe700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90801137e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9070006d40 tx=0x7f9070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9070010840 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9070010e70 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9080113f20 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9070024ea0 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.343+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9080114bc0 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.344+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f907001a4c0 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.344+0000 7f9067fff700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f9068036790 0x7f9068038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.344+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f907004a680 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.344+0000 7f907effd700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f9068036790 0x7f9068038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.344+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90801143b0 con 0x7f9080078ce0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.345+0000 7f907effd700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f9068036790 0x7f9068038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9074009990 tx=0x7f9074006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:12.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.349+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9070018940 con 0x7f9080078ce0 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.569+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f90801001f0 con 0x7f9068036790 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.573+0000 7f9067fff700 1 -- 172.21.15.55:0/2687476221 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f90801001f0 con 0x7f9068036790 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f9068036790 msgr2=0x7f9068038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f9068036790 0x7f9068038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9074009990 tx=0x7f9074006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 msgr2=0x7f90801137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90801137e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9070006d40 tx=0x7f9070009250 comp rx=0 tx=0).stop 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 shutdown_connections 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7f9068036790 0x7f9068038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 --2- 172.21.15.55:0/2687476221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9080078ce0 0x7f90801137e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 >> 172.21.15.55:0/2687476221 conn(0x7f90800fded0 msgr2=0x7f90800ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.575+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 shutdown_connections 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:11.576+0000 7f9085f7d700 1 -- 172.21.15.55:0/2687476221 wait complete. 2024-02-20T07:51:12.127 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying alertmanager service with default placement... 2024-02-20T07:51:12.955 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:12 smithi055 ceph-mon[29254]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:12.955 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:12 smithi055 ceph-mon[29254]: Saving service node-exporter spec with placement * 2024-02-20T07:51:14.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-20T07:51:14.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.316+0000 7fbe37c64700 1 Processor -- start 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.316+0000 7fbe37c64700 1 -- start start 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe37c64700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe37c64700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30101fc0 con 0x7fbe30101670 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe35a00700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe35a00700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50660/0 (socket says 172.21.15.55:50660) 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe35a00700 1 -- 172.21.15.55:0/3412887690 learned_addr learned my addr 172.21.15.55:0/3412887690 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe35a00700 1 -- 172.21.15.55:0/3412887690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30102100 con 0x7fbe30101670 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe35a00700 1 --2- 172.21.15.55:0/3412887690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30101a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0098e0 tx=0x7fbe2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b3d52b5a123a74 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe349fe700 1 -- 172.21.15.55:0/3412887690 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c00b860 con 0x7fbe30101670 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe349fe700 1 -- 172.21.15.55:0/3412887690 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe2c00be90 con 0x7fbe30101670 2024-02-20T07:51:14.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.317+0000 7fbe349fe700 1 -- 172.21.15.55:0/3412887690 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c0103c0 con 0x7fbe30101670 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 -- 172.21.15.55:0/3412887690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 msgr2=0x7fbe30101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 --2- 172.21.15.55:0/3412887690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30101a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0098e0 tx=0x7fbe2c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 -- 172.21.15.55:0/3412887690 shutdown_connections 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 --2- 172.21.15.55:0/3412887690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30101a80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 -- 172.21.15.55:0/3412887690 >> 172.21.15.55:0/3412887690 conn(0x7fbe300fcdd0 msgr2=0x7fbe300ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 -- 172.21.15.55:0/3412887690 shutdown_connections 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.318+0000 7fbe37c64700 1 -- 172.21.15.55:0/3412887690 wait complete. 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe37c64700 1 Processor -- start 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe37c64700 1 -- start start 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe37c64700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe37c64700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30076bf0 con 0x7fbe30101670 2024-02-20T07:51:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe35a00700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe35a00700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30114580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50670/0 (socket says 172.21.15.55:50670) 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe35a00700 1 -- 172.21.15.55:0/2757010052 learned_addr learned my addr 172.21.15.55:0/2757010052 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.319+0000 7fbe35a00700 1 -- 172.21.15.55:0/2757010052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c0094f0 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.320+0000 7fbe35a00700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30114580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c006b00 tx=0x7fbe2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.320+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c0109a0 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.320+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe2c01a430 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.320+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe30076df0 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.320+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c0245b0 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.320+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe30114c80 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.321+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fbe2c024830 con 0x7fbe30101670 2024-02-20T07:51:14.131 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.321+0000 7fbe267fc700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fbe1c036790 0x7fbe1c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.321+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbe2c04a6d0 con 0x7fbe30101670 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.321+0000 7fbe351ff700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fbe1c036790 0x7fbe1c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.321+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3010a590 con 0x7fbe30101670 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.322+0000 7fbe351ff700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fbe1c036790 0x7fbe1c038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009990 tx=0x7fbe20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.326+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbe2c0189a0 con 0x7fbe30101670 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.529+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe30103590 con 0x7fbe1c036790 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.535+0000 7fbe267fc700 1 -- 172.21.15.55:0/2757010052 <== mgr.14118 v2:172.21.15.55:6800/4027903862 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fbe30103590 con 0x7fbe1c036790 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fbe1c036790 msgr2=0x7fbe1c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fbe1c036790 0x7fbe1c038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009990 tx=0x7fbe20006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 msgr2=0x7fbe30114580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30114580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c006b00 tx=0x7fbe2c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:14.132 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 shutdown_connections 2024-02-20T07:51:14.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fbe1c036790 0x7fbe1c038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:14.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 --2- 172.21.15.55:0/2757010052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe30101670 0x7fbe30114580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:14.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 >> 172.21.15.55:0/2757010052 conn(0x7fbe300fcdd0 msgr2=0x7fbe300fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:14.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 shutdown_connections 2024-02-20T07:51:14.133 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:13.537+0000 7fbe37c64700 1 -- 172.21.15.55:0/2757010052 wait complete. 2024-02-20T07:51:14.756 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:14 smithi055 ceph-mon[29254]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:14.756 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:14 smithi055 ceph-mon[29254]: Saving service alertmanager spec with placement count:1 2024-02-20T07:51:14.757 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:14 smithi055 ceph-mon[29254]: from='mgr.14118 172.21.15.55:0/1826130514' entity='mgr.smithi055.stjper' 2024-02-20T07:51:14.757 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:14 smithi055 ceph-mon[29254]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:15.270 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51febb700 1 Processor -- start 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51febb700 1 -- start start 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51febb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa518077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51febb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518079130 con 0x7fa518078d20 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51dc57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa518077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51dc57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa518077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50676/0 (socket says 172.21.15.55:50676) 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51dc57700 1 -- 172.21.15.55:0/1977802600 learned_addr learned my addr 172.21.15.55:0/1977802600 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.533+0000 7fa51dc57700 1 -- 172.21.15.55:0/1977802600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518077770 con 0x7fa518078d20 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51dc57700 1 --2- 172.21.15.55:0/1977802600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa518077230 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098e0 tx=0x7fa514009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96c027d3e8270b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51cc55700 1 -- 172.21.15.55:0/1977802600 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51400b860 con 0x7fa518078d20 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51cc55700 1 -- 172.21.15.55:0/1977802600 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa51400be90 con 0x7fa518078d20 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51cc55700 1 -- 172.21.15.55:0/1977802600 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5140103c0 con 0x7fa518078d20 2024-02-20T07:51:15.271 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51febb700 1 -- 172.21.15.55:0/1977802600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 msgr2=0x7fa518077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51febb700 1 --2- 172.21.15.55:0/1977802600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa518077230 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098e0 tx=0x7fa514009250 comp rx=0 tx=0).stop 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51febb700 1 -- 172.21.15.55:0/1977802600 shutdown_connections 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51febb700 1 --2- 172.21.15.55:0/1977802600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa518077230 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51febb700 1 -- 172.21.15.55:0/1977802600 >> 172.21.15.55:0/1977802600 conn(0x7fa5180fded0 msgr2=0x7fa518100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.534+0000 7fa51febb700 1 -- 172.21.15.55:0/1977802600 shutdown_connections 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51febb700 1 -- 172.21.15.55:0/1977802600 wait complete. 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51febb700 1 Processor -- start 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51febb700 1 -- start start 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51febb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa5181137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51febb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518113d10 con 0x7fa518078d20 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51dc57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa5181137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51dc57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa5181137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50680/0 (socket says 172.21.15.55:50680) 2024-02-20T07:51:15.272 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.535+0000 7fa51dc57700 1 -- 172.21.15.55:0/1169000894 learned_addr learned my addr 172.21.15.55:0/1169000894 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.536+0000 7fa51dc57700 1 -- 172.21.15.55:0/1169000894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5140094f0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.536+0000 7fa51dc57700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa5181137d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa5140094c0 tx=0x7fa514009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.539+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514010890 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.539+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518113f10 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.539+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa514010ec0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.539+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5140243f0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.539+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa518114bb0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.540+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa51402d3f0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.540+0000 7fa50e7fc700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa5040367e0 0x7fa504038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.540+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa51404b2d0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.540+0000 7fa51d456700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa5040367e0 0x7fa504038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.540+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5181143a0 con 0x7fa518078d20 2024-02-20T07:51:15.273 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.540+0000 7fa51d456700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa5040367e0 0x7fa504038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa508009990 tx=0x7fa508006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.545+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa51402b8e0 con 0x7fa518078d20 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.736+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa51805f5c0 con 0x7fa518078d20 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.741+0000 7fa50e7fc700 1 -- 172.21.15.55:0/1169000894 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa51402b2f0 con 0x7fa518078d20 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa5040367e0 msgr2=0x7fa504038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa5040367e0 0x7fa504038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa508009990 tx=0x7fa508006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 msgr2=0x7fa5181137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa5181137d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa5140094c0 tx=0x7fa514009250 comp rx=0 tx=0).stop 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 shutdown_connections 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa5040367e0 0x7fa504038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 --2- 172.21.15.55:0/1169000894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa518078d20 0x7fa5181137d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 >> 172.21.15.55:0/1169000894 conn(0x7fa5180fded0 msgr2=0x7fa5180ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 shutdown_connections 2024-02-20T07:51:15.274 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:14.743+0000 7fa51febb700 1 -- 172.21.15.55:0/1169000894 wait complete. 2024-02-20T07:51:15.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:15 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1169000894' entity='client.admin' 2024-02-20T07:51:16.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc5578c5700 1 Processor -- start 2024-02-20T07:51:16.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc5578c5700 1 -- start start 2024-02-20T07:51:16.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc5578c5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:16.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc5578c5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5501030c0 con 0x7fc550102770 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc555661700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc555661700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50696/0 (socket says 172.21.15.55:50696) 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc555661700 1 -- 172.21.15.55:0/1249537228 learned_addr learned my addr 172.21.15.55:0/1249537228 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc555661700 1 -- 172.21.15.55:0/1249537228 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc550103200 con 0x7fc550102770 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.664+0000 7fc555661700 1 --2- 172.21.15.55:0/1249537228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550102b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0098e0 tx=0x7fc54c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964af1684b34a76a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc547fff700 1 -- 172.21.15.55:0/1249537228 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc54c00b860 con 0x7fc550102770 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc547fff700 1 -- 172.21.15.55:0/1249537228 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc54c00be90 con 0x7fc550102770 2024-02-20T07:51:16.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc547fff700 1 -- 172.21.15.55:0/1249537228 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc54c0103c0 con 0x7fc550102770 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 -- 172.21.15.55:0/1249537228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 msgr2=0x7fc550102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 --2- 172.21.15.55:0/1249537228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550102b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0098e0 tx=0x7fc54c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 -- 172.21.15.55:0/1249537228 shutdown_connections 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 --2- 172.21.15.55:0/1249537228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550102b80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 -- 172.21.15.55:0/1249537228 >> 172.21.15.55:0/1249537228 conn(0x7fc5500fded0 msgr2=0x7fc550100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 -- 172.21.15.55:0/1249537228 shutdown_connections 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.665+0000 7fc5578c5700 1 -- 172.21.15.55:0/1249537228 wait complete. 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc5578c5700 1 Processor -- start 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc5578c5700 1 -- start start 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc5578c5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc5578c5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc550077190 con 0x7fc550102770 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc555661700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc555661700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50698/0 (socket says 172.21.15.55:50698) 2024-02-20T07:51:16.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc555661700 1 -- 172.21.15.55:0/2532214336 learned_addr learned my addr 172.21.15.55:0/2532214336 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc555661700 1 -- 172.21.15.55:0/2532214336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0094f0 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.666+0000 7fc555661700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550078d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0090b0 tx=0x7fc54c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.667+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc54c010840 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.667+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc54c010e70 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.667+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc550077390 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.667+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc54c024e40 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.667+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc550077820 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.668+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc54c01a4e0 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.668+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55004bed0 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.668+0000 7fc545ffb700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fc53c036790 0x7fc53c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.668+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc54c04a8a0 con 0x7fc550102770 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.668+0000 7fc554e60700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fc53c036790 0x7fc53c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.668+0000 7fc554e60700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fc53c036790 0x7fc53c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc540009990 tx=0x7fc540006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:16.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.672+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc54c0109a0 con 0x7fc550102770 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.864+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc550077d60 con 0x7fc550102770 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.869+0000 7fc545ffb700 1 -- 172.21.15.55:0/2532214336 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc54c018350 con 0x7fc550102770 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fc53c036790 msgr2=0x7fc53c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fc53c036790 0x7fc53c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc540009990 tx=0x7fc540006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 msgr2=0x7fc550078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550078d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0090b0 tx=0x7fc54c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 shutdown_connections 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fc53c036790 0x7fc53c038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 --2- 172.21.15.55:0/2532214336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc550102770 0x7fc550078d10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 >> 172.21.15.55:0/2532214336 conn(0x7fc5500fded0 msgr2=0x7fc5500ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 shutdown_connections 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:15.871+0000 7fc5578c5700 1 -- 172.21.15.55:0/2532214336 wait complete. 2024-02-20T07:51:16.472 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling the dashboard module... 2024-02-20T07:51:17.124 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:16 smithi055 ceph-mon[29254]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:17.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:16 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2532214336' entity='client.admin' 2024-02-20T07:51:18.203 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:17 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2539372634' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb77d52b700 1 Processor -- start 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb77d52b700 1 -- start start 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb77d52b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb778102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb77d52b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781030c0 con 0x7fb778102770 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb778102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb778102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50708/0 (socket says 172.21.15.55:50708) 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb776ffd700 1 -- 172.21.15.55:0/3177419395 learned_addr learned my addr 172.21.15.55:0/3177419395 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.867+0000 7fb776ffd700 1 -- 172.21.15.55:0/3177419395 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb778103200 con 0x7fb778102770 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb776ffd700 1 --2- 172.21.15.55:0/3177419395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb778102b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb7680098e0 tx=0x7fb768009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b9acbf9914b052 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb775ffb700 1 -- 172.21.15.55:0/3177419395 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76800b860 con 0x7fb778102770 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb775ffb700 1 -- 172.21.15.55:0/3177419395 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb76800be90 con 0x7fb778102770 2024-02-20T07:51:18.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb775ffb700 1 -- 172.21.15.55:0/3177419395 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7680103c0 con 0x7fb778102770 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb77d52b700 1 -- 172.21.15.55:0/3177419395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 msgr2=0x7fb778102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb77d52b700 1 --2- 172.21.15.55:0/3177419395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb778102b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb7680098e0 tx=0x7fb768009250 comp rx=0 tx=0).stop 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb77d52b700 1 -- 172.21.15.55:0/3177419395 shutdown_connections 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb77d52b700 1 --2- 172.21.15.55:0/3177419395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb778102b80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.868+0000 7fb77d52b700 1 -- 172.21.15.55:0/3177419395 >> 172.21.15.55:0/3177419395 conn(0x7fb7780fded0 msgr2=0x7fb778100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.869+0000 7fb77d52b700 1 -- 172.21.15.55:0/3177419395 shutdown_connections 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.869+0000 7fb77d52b700 1 -- 172.21.15.55:0/3177419395 wait complete. 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.869+0000 7fb77d52b700 1 Processor -- start 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.869+0000 7fb77d52b700 1 -- start start 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.869+0000 7fb77d52b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb7781115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.869+0000 7fb77d52b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb778111b00 con 0x7fb778102770 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.870+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb7781115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:18.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.870+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb7781115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50718/0 (socket says 172.21.15.55:50718) 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.870+0000 7fb776ffd700 1 -- 172.21.15.55:0/2539372634 learned_addr learned my addr 172.21.15.55:0/2539372634 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.870+0000 7fb776ffd700 1 -- 172.21.15.55:0/2539372634 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7680094f0 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.870+0000 7fb776ffd700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb7781115c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb7680090b0 tx=0x7fb768009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.871+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb768010840 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.871+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb778111d00 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.871+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb768010e70 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.871+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb768024e40 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.871+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7781129a0 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.872+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb76801a510 con 0x7fb778102770 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.872+0000 7fb75f7fe700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb760036740 0x7fb760038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:18.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.872+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77805f5c0 con 0x7fb778102770 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.872+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb76804a7e0 con 0x7fb778102770 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.872+0000 7fb7767fc700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb760036740 0x7fb760038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.872+0000 7fb7767fc700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb760036740 0x7fb760038bf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009990 tx=0x7fb76c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:16.877+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb76802e440 con 0x7fb778102770 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.123+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb77804bed0 con 0x7fb778102770 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.878+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fb768024430 con 0x7fb778102770 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.879+0000 7fb75f7fe700 1 -- 172.21.15.55:0/2539372634 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb768018350 con 0x7fb778102770 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb760036740 msgr2=0x7fb760038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb760036740 0x7fb760038bf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009990 tx=0x7fb76c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 msgr2=0x7fb7781115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb7781115c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb7680090b0 tx=0x7fb768009250 comp rx=0 tx=0).stop 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 shutdown_connections 2024-02-20T07:51:18.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fb760036740 0x7fb760038bf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:18.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 --2- 172.21.15.55:0/2539372634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb778102770 0x7fb7781115c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:18.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 >> 172.21.15.55:0/2539372634 conn(0x7fb7780fded0 msgr2=0x7fb7780ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:18.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 shutdown_connections 2024-02-20T07:51:18.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:17.882+0000 7fb77d52b700 1 -- 172.21.15.55:0/2539372634 wait complete. 2024-02-20T07:51:19.124 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:18 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2539372634' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-20T07:51:19.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:18 smithi055 ceph-mon[29254]: mgrmap e9: smithi055.stjper(active, since 24s) 2024-02-20T07:51:19.784 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:51:19.784 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-20T07:51:19.784 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-20T07:51:19.784 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "active_name": "smithi055.stjper", 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.917+0000 7fa2faad7700 1 Processor -- start 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.917+0000 7fa2faad7700 1 -- start start 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2faad7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2faad7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f41030c0 con 0x7fa2f4102770 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2f8873700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2f8873700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50744/0 (socket says 172.21.15.55:50744) 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2f8873700 1 -- 172.21.15.55:0/1571347943 learned_addr learned my addr 172.21.15.55:0/1571347943 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2f8873700 1 -- 172.21.15.55:0/1571347943 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4103200 con 0x7fa2f4102770 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.918+0000 7fa2f8873700 1 --2- 172.21.15.55:0/1571347943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4102b80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80098e0 tx=0x7fa2e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66692f82edb97629 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2f37fe700 1 -- 172.21.15.55:0/1571347943 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e800b860 con 0x7fa2f4102770 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2f37fe700 1 -- 172.21.15.55:0/1571347943 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2e800be90 con 0x7fa2f4102770 2024-02-20T07:51:19.785 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2f37fe700 1 -- 172.21.15.55:0/1571347943 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e80103c0 con 0x7fa2f4102770 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 -- 172.21.15.55:0/1571347943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 msgr2=0x7fa2f4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 --2- 172.21.15.55:0/1571347943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4102b80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80098e0 tx=0x7fa2e8009250 comp rx=0 tx=0).stop 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 -- 172.21.15.55:0/1571347943 shutdown_connections 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 --2- 172.21.15.55:0/1571347943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4102b80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 -- 172.21.15.55:0/1571347943 >> 172.21.15.55:0/1571347943 conn(0x7fa2f40fded0 msgr2=0x7fa2f4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 -- 172.21.15.55:0/1571347943 shutdown_connections 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.919+0000 7fa2faad7700 1 -- 172.21.15.55:0/1571347943 wait complete. 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2faad7700 1 Processor -- start 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2faad7700 1 -- start start 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2faad7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2faad7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4111a80 con 0x7fa2f4102770 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2f8873700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2f8873700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50746/0 (socket says 172.21.15.55:50746) 2024-02-20T07:51:19.786 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.920+0000 7fa2f8873700 1 -- 172.21.15.55:0/514981754 learned_addr learned my addr 172.21.15.55:0/514981754 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2f8873700 1 -- 172.21.15.55:0/514981754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80094f0 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2f8873700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4111540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8006b00 tx=0x7fa2e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e80109a0 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2e801a430 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f4111c80 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e80245b0 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.921+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f4112920 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.922+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa2e80247e0 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.922+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f405f5c0 con 0x7fa2f4102770 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.922+0000 7fa2f17fa700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 0x7fa2e4038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:19.787 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.922+0000 7fa2f3fff700 1 -- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 msgr2=0x7fa2e4038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.922+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa2e804a910 con 0x7fa2f4102770 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.922+0000 7fa2f3fff700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 0x7fa2e4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:18.927+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa2e8018530 con 0x7fa2f4102770 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.123+0000 7fa2f3fff700 1 -- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 msgr2=0x7fa2e4038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.123+0000 7fa2f3fff700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 0x7fa2e4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.177+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa2f4112c60 con 0x7fa2f4102770 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.177+0000 7fa2f17fa700 1 -- 172.21.15.55:0/514981754 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa2e8023e40 con 0x7fa2f4102770 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.180+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 msgr2=0x7fa2e4038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.180+0000 7fa2faad7700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 0x7fa2e4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.180+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 msgr2=0x7fa2f4111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.180+0000 7fa2faad7700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4111540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8006b00 tx=0x7fa2e8009250 comp rx=0 tx=0).stop 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.181+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 shutdown_connections 2024-02-20T07:51:19.788 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.181+0000 7fa2faad7700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7fa2e4036830 0x7fa2e4038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:19.789 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.181+0000 7fa2faad7700 1 --2- 172.21.15.55:0/514981754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f4102770 0x7fa2f4111540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:19.789 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.181+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 >> 172.21.15.55:0/514981754 conn(0x7fa2f40fded0 msgr2=0x7fa2f40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:19.789 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.181+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 shutdown_connections 2024-02-20T07:51:19.789 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:19.181+0000 7fa2faad7700 1 -- 172.21.15.55:0/514981754 wait complete. 2024-02-20T07:51:19.789 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for the mgr to restart... 2024-02-20T07:51:19.789 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr epoch 9... 2024-02-20T07:51:20.066 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:19 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/514981754' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T07:51:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: Active manager daemon smithi055.stjper restarted 2024-02-20T07:51:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: Activating manager daemon smithi055.stjper 2024-02-20T07:51:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: osdmap e3: 0 total, 0 up, 0 in 2024-02-20T07:51:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: mgrmap e10: smithi055.stjper(active, starting, since 0.00664626s) 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr metadata", "who": "smithi055.stjper", "id": "smithi055.stjper"}]: dispatch 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: Manager daemon smithi055.stjper is now available 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:24 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:25 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:51:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:25 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/trash_purge_schedule"}]: dispatch 2024-02-20T07:51:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:25 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:25 smithi055 ceph-mon[29254]: mgrmap e11: smithi055.stjper(active, since 1.01157s) 2024-02-20T07:51:26.421 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:51:26.421 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.186+0000 7ff1766d6700 1 Processor -- start 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.186+0000 7ff1766d6700 1 -- start start 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.186+0000 7ff1766d6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff170105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.186+0000 7ff1766d6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff170105960 con 0x7ff170076b80 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff170105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff170105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50752/0 (socket says 172.21.15.55:50752) 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16ffff700 1 -- 172.21.15.55:0/2951914410 learned_addr learned my addr 172.21.15.55:0/2951914410 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16ffff700 1 -- 172.21.15.55:0/2951914410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff170105aa0 con 0x7ff170076b80 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16ffff700 1 --2- 172.21.15.55:0/2951914410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff170105420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff1600098e0 tx=0x7ff160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d305b14ee96416c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16effd700 1 -- 172.21.15.55:0/2951914410 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff16000b860 con 0x7ff170076b80 2024-02-20T07:51:26.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16effd700 1 -- 172.21.15.55:0/2951914410 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff16000be90 con 0x7ff170076b80 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.187+0000 7ff16effd700 1 -- 172.21.15.55:0/2951914410 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1600103c0 con 0x7ff170076b80 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 -- 172.21.15.55:0/2951914410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 msgr2=0x7ff170105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 --2- 172.21.15.55:0/2951914410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff170105420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff1600098e0 tx=0x7ff160009250 comp rx=0 tx=0).stop 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 -- 172.21.15.55:0/2951914410 shutdown_connections 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 --2- 172.21.15.55:0/2951914410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff170105420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 -- 172.21.15.55:0/2951914410 >> 172.21.15.55:0/2951914410 conn(0x7ff1700fcdd0 msgr2=0x7ff1700ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 -- 172.21.15.55:0/2951914410 shutdown_connections 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.188+0000 7ff1766d6700 1 -- 172.21.15.55:0/2951914410 wait complete. 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff1766d6700 1 Processor -- start 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff1766d6700 1 -- start start 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff1766d6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff1701115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff1766d6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff170111af0 con 0x7ff170076b80 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff16ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff1701115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff16ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff1701115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50768/0 (socket says 172.21.15.55:50768) 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.189+0000 7ff16ffff700 1 -- 172.21.15.55:0/1226157670 learned_addr learned my addr 172.21.15.55:0/1226157670 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff16ffff700 1 -- 172.21.15.55:0/1226157670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1600094f0 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff16ffff700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff1701115b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1600090b0 tx=0x7ff160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff160010890 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff170111cf0 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff160010ec0 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff160024e70 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.190+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff170112990 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.191+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7ff16001a550 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.191+0000 7ff16cff9700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.191+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 --> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff17005f5c0 con 0x7ff1580367e0 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.191+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff16004a880 con 0x7ff170076b80 2024-02-20T07:51:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.191+0000 7ff16f7fe700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 msgr2=0x7ff158038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.191+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.392+0000 7ff16f7fe700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 msgr2=0x7ff158038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.392+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.793+0000 7ff16f7fe700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 msgr2=0x7ff158038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:20.793+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:21.594+0000 7ff16f7fe700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 msgr2=0x7ff158038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:21.594+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:23.195+0000 7ff16f7fe700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 msgr2=0x7ff158038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/4027903862 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:23.195+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:24.636+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7ff160024430 con 0x7ff170076b80 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:24.636+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 msgr2=0x7ff158038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:24.636+0000 7ff16cff9700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.640+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7ff160024750 con 0x7ff170076b80 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.641+0000 7ff16cff9700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7ff158039ff0 0x7ff15803c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:26.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.641+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff17005f5c0 con 0x7ff158039ff0 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.641+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7ff158039ff0 0x7ff15803c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.642+0000 7ff16f7fe700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7ff158039ff0 0x7ff15803c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff1640039d0 tx=0x7ff1640092f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.642+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7ff17005f5c0 con 0x7ff158039ff0 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.648+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff170112180 con 0x7ff158039ff0 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.648+0000 7ff16cff9700 1 -- 172.21.15.55:0/1226157670 <== mgr.14160 v2:172.21.15.55:6800/2284801197 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff170112180 con 0x7ff158039ff0 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7ff158039ff0 msgr2=0x7ff15803c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7ff158039ff0 0x7ff15803c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff1640039d0 tx=0x7ff1640092f0 comp rx=0 tx=0).stop 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 msgr2=0x7ff1701115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff1701115b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1600090b0 tx=0x7ff160009250 comp rx=0 tx=0).stop 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 shutdown_connections 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7ff158039ff0 0x7ff15803c3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff170076b80 0x7ff1701115b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 --2- 172.21.15.55:0/1226157670 >> [v2:172.21.15.55:6800/4027903862,v1:172.21.15.55:6801/4027903862] conn(0x7ff1580367e0 0x7ff158038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:26.426 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 >> 172.21.15.55:0/1226157670 conn(0x7ff1700fcdd0 msgr2=0x7ff1700fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:26.427 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 shutdown_connections 2024-02-20T07:51:26.427 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:25.649+0000 7ff1766d6700 1 -- 172.21.15.55:0/1226157670 wait complete. 2024-02-20T07:51:26.427 INFO:teuthology.orchestra.run.smithi055.stdout:mgr epoch 9 is available 2024-02-20T07:51:26.427 INFO:teuthology.orchestra.run.smithi055.stdout:Generating a dashboard self-signed certificate... 2024-02-20T07:51:26.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:26 smithi055 ceph-mon[29254]: [20/Feb/2024:07:51:25] ENGINE Bus STARTING 2024-02-20T07:51:26.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:26 smithi055 ceph-mon[29254]: [20/Feb/2024:07:51:25] ENGINE Serving on https://172.21.15.55:7150 2024-02-20T07:51:26.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:26 smithi055 ceph-mon[29254]: [20/Feb/2024:07:51:25] ENGINE Bus STARTED 2024-02-20T07:51:26.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:26 smithi055 ceph-mon[29254]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T07:51:26.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:26 smithi055 ceph-mon[29254]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T07:51:27.666 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-20T07:51:27.666 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.765+0000 7f5f2da60700 1 Processor -- start 2024-02-20T07:51:27.666 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f2da60700 1 -- start start 2024-02-20T07:51:27.666 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f2da60700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f280760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:27.666 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f2da60700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28077fc0 con 0x7f5f28077bb0 2024-02-20T07:51:27.666 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f26ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f280760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f26ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f280760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58338/0 (socket says 172.21.15.55:58338) 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f26ffd700 1 -- 172.21.15.55:0/3277973399 learned_addr learned my addr 172.21.15.55:0/3277973399 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f26ffd700 1 -- 172.21.15.55:0/3277973399 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28076600 con 0x7f5f28077bb0 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.766+0000 7f5f26ffd700 1 --2- 172.21.15.55:0/3277973399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f280760c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f180098e0 tx=0x7f5f18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d947c1b286dfc476 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f25ffb700 1 -- 172.21.15.55:0/3277973399 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1800b860 con 0x7f5f28077bb0 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f25ffb700 1 -- 172.21.15.55:0/3277973399 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f1800be90 con 0x7f5f28077bb0 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f25ffb700 1 -- 172.21.15.55:0/3277973399 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f180103c0 con 0x7f5f28077bb0 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 -- 172.21.15.55:0/3277973399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 msgr2=0x7f5f280760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 --2- 172.21.15.55:0/3277973399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f280760c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f180098e0 tx=0x7f5f18009250 comp rx=0 tx=0).stop 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 -- 172.21.15.55:0/3277973399 shutdown_connections 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 --2- 172.21.15.55:0/3277973399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f280760c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 -- 172.21.15.55:0/3277973399 >> 172.21.15.55:0/3277973399 conn(0x7f5f280fded0 msgr2=0x7f5f28100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 -- 172.21.15.55:0/3277973399 shutdown_connections 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.767+0000 7f5f2da60700 1 -- 172.21.15.55:0/3277973399 wait complete. 2024-02-20T07:51:27.667 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.768+0000 7f5f2da60700 1 Processor -- start 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.768+0000 7f5f2da60700 1 -- start start 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f2da60700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f281137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f2da60700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28113d10 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f26ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f281137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f26ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f281137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58354/0 (socket says 172.21.15.55:58354) 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f26ffd700 1 -- 172.21.15.55:0/2070310349 learned_addr learned my addr 172.21.15.55:0/2070310349 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f26ffd700 1 -- 172.21.15.55:0/2070310349 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f180094f0 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f26ffd700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f281137d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f18006d40 tx=0x7f5f18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f18010840 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f18010e70 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.769+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f28113f10 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.770+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f18024ea0 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.770+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f28114bb0 con 0x7f5f28077bb0 2024-02-20T07:51:27.668 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.771+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f5f1801a4c0 con 0x7f5f28077bb0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.771+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2805f5c0 con 0x7f5f28077bb0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.771+0000 7f5f0f7fe700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f5f100367e0 0x7f5f10038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.771+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5f1804b870 con 0x7f5f28077bb0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.771+0000 7f5f267fc700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f5f100367e0 0x7f5f10038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.771+0000 7f5f267fc700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f5f100367e0 0x7f5f10038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009990 tx=0x7f5f1c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.776+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5f18018940 con 0x7f5f28077bb0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:26.991+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f28100210 con 0x7f5f100367e0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.061+0000 7f5f0f7fe700 1 -- 172.21.15.55:0/2070310349 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5f28100210 con 0x7f5f100367e0 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.063+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f5f100367e0 msgr2=0x7f5f10038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.063+0000 7f5f2da60700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f5f100367e0 0x7f5f10038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009990 tx=0x7f5f1c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.063+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 msgr2=0x7f5f281137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.063+0000 7f5f2da60700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f281137d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f18006d40 tx=0x7f5f18009250 comp rx=0 tx=0).stop 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.064+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 shutdown_connections 2024-02-20T07:51:27.669 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.064+0000 7f5f2da60700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f5f100367e0 0x7f5f10038c90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:27.670 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.064+0000 7f5f2da60700 1 --2- 172.21.15.55:0/2070310349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f28077bb0 0x7f5f281137d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:27.670 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.064+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 >> 172.21.15.55:0/2070310349 conn(0x7f5f280fded0 msgr2=0x7f5f280ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:27.670 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.064+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 shutdown_connections 2024-02-20T07:51:27.670 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:27.064+0000 7f5f2da60700 1 -- 172.21.15.55:0/2070310349 wait complete. 2024-02-20T07:51:27.670 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial admin user... 2024-02-20T07:51:27.787 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:27 smithi055 ceph-mon[29254]: mgrmap e12: smithi055.stjper(active, since 2s) 2024-02-20T07:51:27.787 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:27 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:27.787 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:27 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:27.787 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:27 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:29.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:28 smithi055 ceph-mon[29254]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:29.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:28 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:29.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:28 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$CbAKlqQOm3ggERXfQSrfJeSa54G/RXk/Q1BGZDJH/PuJ/2eAdoCFK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708415488, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f735d78c700 1 Processor -- start 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f735d78c700 1 -- start start 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f735d78c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f7358102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f735d78c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73581030c0 con 0x7f7358102770 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f7356ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f7358102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f7356ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f7358102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58368/0 (socket says 172.21.15.55:58368) 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f7356ffd700 1 -- 172.21.15.55:0/1666308665 learned_addr learned my addr 172.21.15.55:0/1666308665 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.062+0000 7f7356ffd700 1 -- 172.21.15.55:0/1666308665 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7358103200 con 0x7f7358102770 2024-02-20T07:51:29.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f7356ffd700 1 --2- 172.21.15.55:0/1666308665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f7358102b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f73480098e0 tx=0x7f7348009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed523effb02a1d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f7355ffb700 1 -- 172.21.15.55:0/1666308665 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f734800b860 con 0x7f7358102770 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f7355ffb700 1 -- 172.21.15.55:0/1666308665 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f734800be90 con 0x7f7358102770 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f7355ffb700 1 -- 172.21.15.55:0/1666308665 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73480103c0 con 0x7f7358102770 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 -- 172.21.15.55:0/1666308665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 msgr2=0x7f7358102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 --2- 172.21.15.55:0/1666308665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f7358102b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f73480098e0 tx=0x7f7348009250 comp rx=0 tx=0).stop 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 -- 172.21.15.55:0/1666308665 shutdown_connections 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 --2- 172.21.15.55:0/1666308665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f7358102b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 -- 172.21.15.55:0/1666308665 >> 172.21.15.55:0/1666308665 conn(0x7f73580fded0 msgr2=0x7f7358100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 -- 172.21.15.55:0/1666308665 shutdown_connections 2024-02-20T07:51:29.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.063+0000 7f735d78c700 1 -- 172.21.15.55:0/1666308665 wait complete. 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.064+0000 7f735d78c700 1 Processor -- start 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.064+0000 7f735d78c700 1 -- start start 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.064+0000 7f735d78c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f73581115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.064+0000 7f735d78c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7358111af0 con 0x7f7358102770 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.064+0000 7f7356ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f73581115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f7356ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f73581115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58374/0 (socket says 172.21.15.55:58374) 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f7356ffd700 1 -- 172.21.15.55:0/574602500 learned_addr learned my addr 172.21.15.55:0/574602500 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f7356ffd700 1 -- 172.21.15.55:0/574602500 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73480094f0 con 0x7f7358102770 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f7356ffd700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f73581115b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7348011680 tx=0x7f7348009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:29.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7348010890 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7358111cf0 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7348010ec0 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73480243f0 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.065+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7358112990 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.066+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f734802d3f0 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.066+0000 7f733f7fe700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f73400367e0 0x7f7340038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.066+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735805f5c0 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.066+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f734804a510 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.066+0000 7f73567fc700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f73400367e0 0x7f7340038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.067+0000 7f73567fc700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f73400367e0 0x7f7340038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f734c009990 tx=0x7f734c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.072+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f734802b8e0 con 0x7f7358102770 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.285+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f735805f3e0 con 0x7f73400367e0 2024-02-20T07:51:29.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.536+0000 7f733f7fe700 1 -- 172.21.15.55:0/574602500 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f735805f3e0 con 0x7f73400367e0 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f73400367e0 msgr2=0x7f7340038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f73400367e0 0x7f7340038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f734c009990 tx=0x7f734c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 msgr2=0x7f73581115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f73581115b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7348011680 tx=0x7f7348009250 comp rx=0 tx=0).stop 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 shutdown_connections 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f73400367e0 0x7f7340038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 --2- 172.21.15.55:0/574602500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7358102770 0x7f73581115b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 >> 172.21.15.55:0/574602500 conn(0x7f73580fded0 msgr2=0x7f73580ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 shutdown_connections 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:28.538+0000 7f735d78c700 1 -- 172.21.15.55:0/574602500 wait complete. 2024-02-20T07:51:29.473 INFO:teuthology.orchestra.run.smithi055.stdout:Fetching dashboard port number... 2024-02-20T07:51:29.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:29 smithi055 ceph-mon[29254]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:30.924 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 8443 2024-02-20T07:51:30.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.929+0000 7fa03daea700 1 Processor -- start 2024-02-20T07:51:30.929 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.929+0000 7fa03daea700 1 -- start start 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.929+0000 7fa03daea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa038102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.929+0000 7fa03daea700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381030c0 con 0x7fa038102770 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0377fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa038102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0377fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa038102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58384/0 (socket says 172.21.15.55:58384) 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0377fe700 1 -- 172.21.15.55:0/4003920683 learned_addr learned my addr 172.21.15.55:0/4003920683 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0377fe700 1 -- 172.21.15.55:0/4003920683 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa038103200 con 0x7fa038102770 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0377fe700 1 --2- 172.21.15.55:0/4003920683 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa038102b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0098e0 tx=0x7fa02c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2dae629971723a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0367fc700 1 -- 172.21.15.55:0/4003920683 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c00b860 con 0x7fa038102770 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0367fc700 1 -- 172.21.15.55:0/4003920683 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa02c00be90 con 0x7fa038102770 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.930+0000 7fa0367fc700 1 -- 172.21.15.55:0/4003920683 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c0103c0 con 0x7fa038102770 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 -- 172.21.15.55:0/4003920683 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 msgr2=0x7fa038102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 --2- 172.21.15.55:0/4003920683 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa038102b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0098e0 tx=0x7fa02c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 -- 172.21.15.55:0/4003920683 shutdown_connections 2024-02-20T07:51:30.930 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 --2- 172.21.15.55:0/4003920683 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa038102b80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 -- 172.21.15.55:0/4003920683 >> 172.21.15.55:0/4003920683 conn(0x7fa0380fded0 msgr2=0x7fa038100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 -- 172.21.15.55:0/4003920683 shutdown_connections 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 -- 172.21.15.55:0/4003920683 wait complete. 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.931+0000 7fa03daea700 1 Processor -- start 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa03daea700 1 -- start start 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa03daea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa0381115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa03daea700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa038111b00 con 0x7fa038102770 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa0377fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa0381115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa0377fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa0381115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58392/0 (socket says 172.21.15.55:58392) 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa0377fe700 1 -- 172.21.15.55:0/3945395277 learned_addr learned my addr 172.21.15.55:0/3945395277 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa0377fe700 1 -- 172.21.15.55:0/3945395277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0094f0 con 0x7fa038102770 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa0377fe700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa0381115c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa02c011680 tx=0x7fa02c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c010890 con 0x7fa038102770 2024-02-20T07:51:30.931 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.932+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa038111d00 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.933+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa02c010ec0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.933+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c0243f0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.933+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0381129a0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.934+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa02c01a8c0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.934+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03805f5c0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.934+0000 7fa01ffff700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa0200367e0 0x7fa020038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.934+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa02c04a360 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.934+0000 7fa036ffd700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa0200367e0 0x7fa020038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.934+0000 7fa036ffd700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa0200367e0 0x7fa020038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa028009990 tx=0x7fa028006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:29.938+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa02c006d70 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.134+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa03804bed0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.134+0000 7fa01ffff700 1 -- 172.21.15.55:0/3945395277 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa02c02b2f0 con 0x7fa038102770 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa0200367e0 msgr2=0x7fa020038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:30.932 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa0200367e0 0x7fa020038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa028009990 tx=0x7fa028006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 msgr2=0x7fa0381115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa0381115c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa02c011680 tx=0x7fa02c009250 comp rx=0 tx=0).stop 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 shutdown_connections 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa0200367e0 0x7fa020038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 --2- 172.21.15.55:0/3945395277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa038102770 0x7fa0381115c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.136+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 >> 172.21.15.55:0/3945395277 conn(0x7fa0380fded0 msgr2=0x7fa0380ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.137+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 shutdown_connections 2024-02-20T07:51:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:30.137+0000 7fa03daea700 1 -- 172.21.15.55:0/3945395277 wait complete. 2024-02-20T07:51:30.937 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:51:30.937 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-02-20T07:51:30.945 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:51:30.945 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-02-20T07:51:30.945 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-02-20T07:51:30.945 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout:Ceph Dashboard is now available at: 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout: URL: https://smithi055.front.sepia.ceph.com:8443/ 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout: User: admin 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout: Password: f1inx95yqj 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:30.947 INFO:teuthology.orchestra.run.smithi055.stdout:Saving cluster configuration to /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config directory 2024-02-20T07:51:30.948 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling autotune for osd_memory_target 2024-02-20T07:51:31.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:30 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3945395277' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-20T07:51:32.085 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:31 smithi055 ceph-mon[29254]: mgrmap e13: smithi055.stjper(active, since 6s) 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.364+0000 7f681c265700 1 Processor -- start 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681c265700 1 -- start start 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681c265700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681c265700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68141058a0 con 0x7f68140714a0 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681a001700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681a001700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58400/0 (socket says 172.21.15.55:58400) 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681a001700 1 -- 172.21.15.55:0/3576954560 learned_addr learned my addr 172.21.15.55:0/3576954560 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:32.093 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.365+0000 7f681a001700 1 -- 172.21.15.55:0/3576954560 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68141059e0 con 0x7f68140714a0 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.366+0000 7f681a001700 1 --2- 172.21.15.55:0/3576954560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814105360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f680c009e50 tx=0x7f680c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9148e3ab4b94f2a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.366+0000 7f6818fff700 1 -- 172.21.15.55:0/3576954560 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c0092e0 con 0x7f68140714a0 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.366+0000 7f6818fff700 1 -- 172.21.15.55:0/3576954560 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f680c012430 con 0x7f68140714a0 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.366+0000 7f6818fff700 1 -- 172.21.15.55:0/3576954560 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c0075a0 con 0x7f68140714a0 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.366+0000 7f681c265700 1 -- 172.21.15.55:0/3576954560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 msgr2=0x7f6814105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.366+0000 7f681c265700 1 --2- 172.21.15.55:0/3576954560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814105360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f680c009e50 tx=0x7f680c009840 comp rx=0 tx=0).stop 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 -- 172.21.15.55:0/3576954560 shutdown_connections 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 --2- 172.21.15.55:0/3576954560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814105360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 -- 172.21.15.55:0/3576954560 >> 172.21.15.55:0/3576954560 conn(0x7f68140fcdd0 msgr2=0x7f68140ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 -- 172.21.15.55:0/3576954560 shutdown_connections 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 -- 172.21.15.55:0/3576954560 wait complete. 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 Processor -- start 2024-02-20T07:51:32.094 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 -- start start 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.367+0000 7f681c265700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6814101260 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681a001700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681a001700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814102de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58414/0 (socket says 172.21.15.55:58414) 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681a001700 1 -- 172.21.15.55:0/3165680313 learned_addr learned my addr 172.21.15.55:0/3165680313 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681a001700 1 -- 172.21.15.55:0/3165680313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c009ae0 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681a001700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814102de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f680c012de0 tx=0x7f680c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c0092e0 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f680c007d30 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6814101460 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c025a00 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.368+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68141018f0 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.369+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f680c019050 con 0x7f68140714a0 2024-02-20T07:51:32.095 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.369+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681410b680 con 0x7f68140714a0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.369+0000 7f68127fc700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f67fc036880 0x7f67fc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.369+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f680c04a280 con 0x7f68140714a0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.369+0000 7f6819800700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f67fc036880 0x7f67fc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.370+0000 7f6819800700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f67fc036880 0x7f67fc038d30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6808009990 tx=0x7f6808006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.375+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f680c02d9a0 con 0x7f68140714a0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.567+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f68141067c0 con 0x7f68140714a0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.568+0000 7f68127fc700 1 -- 172.21.15.55:0/3165680313 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f680c017070 con 0x7f68140714a0 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.569+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f67fc036880 msgr2=0x7f67fc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.569+0000 7f681c265700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f67fc036880 0x7f67fc038d30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6808009990 tx=0x7f6808006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.569+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 msgr2=0x7f6814102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.569+0000 7f681c265700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814102de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f680c012de0 tx=0x7f680c009840 comp rx=0 tx=0).stop 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.570+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 shutdown_connections 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.570+0000 7f681c265700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f67fc036880 0x7f67fc038d30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:32.096 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.570+0000 7f681c265700 1 --2- 172.21.15.55:0/3165680313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68140714a0 0x7f6814102de0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:32.097 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.570+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 >> 172.21.15.55:0/3165680313 conn(0x7f68140fcdd0 msgr2=0x7f68140fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:32.097 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.570+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 shutdown_connections 2024-02-20T07:51:32.097 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:31.570+0000 7f681c265700 1 -- 172.21.15.55:0/3165680313 wait complete. 2024-02-20T07:51:33.359 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5cade700 1 Processor -- start 2024-02-20T07:51:33.359 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5cade700 1 -- start start 2024-02-20T07:51:33.359 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5cade700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5cade700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c58079130 con 0x7f6c58078d20 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54836/0 (socket says 172.21.15.55:54836) 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.497+0000 7f6c5659c700 1 -- 172.21.15.55:0/3656910867 learned_addr learned my addr 172.21.15.55:0/3656910867 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5659c700 1 -- 172.21.15.55:0/3656910867 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c58077770 con 0x7f6c58078d20 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5659c700 1 --2- 172.21.15.55:0/3656910867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58077230 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c480098e0 tx=0x7f6c48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aace3e14e96b11c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5559a700 1 -- 172.21.15.55:0/3656910867 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c4800b860 con 0x7f6c58078d20 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5559a700 1 -- 172.21.15.55:0/3656910867 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c4800be90 con 0x7f6c58078d20 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5559a700 1 -- 172.21.15.55:0/3656910867 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c480103c0 con 0x7f6c58078d20 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5cade700 1 -- 172.21.15.55:0/3656910867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 msgr2=0x7f6c58077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.498+0000 7f6c5cade700 1 --2- 172.21.15.55:0/3656910867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58077230 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c480098e0 tx=0x7f6c48009250 comp rx=0 tx=0).stop 2024-02-20T07:51:33.360 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 -- 172.21.15.55:0/3656910867 shutdown_connections 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 --2- 172.21.15.55:0/3656910867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58077230 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 -- 172.21.15.55:0/3656910867 >> 172.21.15.55:0/3656910867 conn(0x7f6c580fded0 msgr2=0x7f6c58100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 -- 172.21.15.55:0/3656910867 shutdown_connections 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 -- 172.21.15.55:0/3656910867 wait complete. 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 Processor -- start 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 -- start start 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5cade700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c58113d50 con 0x7f6c58078d20 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54840/0 (socket says 172.21.15.55:54840) 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.499+0000 7f6c5659c700 1 -- 172.21.15.55:0/3169318372 learned_addr learned my addr 172.21.15.55:0/3169318372 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c5659c700 1 -- 172.21.15.55:0/3169318372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480094f0 con 0x7f6c58078d20 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c5659c700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58113810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6c480094c0 tx=0x7f6c48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:33.361 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c48010890 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c48010ec0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c58113f50 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c480243f0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.500+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c58114bf0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.501+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6c4801a7e0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.501+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5805f5c0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.501+0000 7f6c46ffd700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6c400368d0 0x7f6c40038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.501+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6c4804a440 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.501+0000 7f6c55d9b700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6c400368d0 0x7f6c40038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.501+0000 7f6c55d9b700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6c400368d0 0x7f6c40038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009990 tx=0x7f6c4c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.506+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6c4802b8c0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.807+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f6c5804bed0 con 0x7f6c58078d20 2024-02-20T07:51:33.362 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.814+0000 7f6c46ffd700 1 -- 172.21.15.55:0/3169318372 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f6c4802b2f0 con 0x7f6c58078d20 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6c400368d0 msgr2=0x7f6c40038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6c400368d0 0x7f6c40038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009990 tx=0x7f6c4c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 msgr2=0x7f6c58113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58113810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6c480094c0 tx=0x7f6c48009250 comp rx=0 tx=0).stop 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 shutdown_connections 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6c400368d0 0x7f6c40038d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 --2- 172.21.15.55:0/3169318372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c58078d20 0x7f6c58113810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 >> 172.21.15.55:0/3169318372 conn(0x7f6c580fded0 msgr2=0x7f6c580ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 shutdown_connections 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2024-02-20T07:51:32.816+0000 7f6c5cade700 1 -- 172.21.15.55:0/3169318372 wait complete. 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.363 INFO:teuthology.orchestra.run.smithi055.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout:Or, if you are only running a single cluster on this host: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: ceph telemetry on 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout:For more information see: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:33.364 INFO:teuthology.orchestra.run.smithi055.stdout:Bootstrap complete. 2024-02-20T07:51:33.395 INFO:tasks.cephadm:Fetching config... 2024-02-20T07:51:33.395 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:51:33.395 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-20T07:51:33.410 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-20T07:51:33.410 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:51:33.410 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-20T07:51:33.465 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-20T07:51:33.465 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:51:33.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/keyring of=/dev/stdout 2024-02-20T07:51:33.532 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-20T07:51:33.533 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:51:33.533 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-20T07:51:33.589 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-20T07:51:33.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuJvGIPV7FFdFaQZp6plHTpj3SLxFP3boKwhAWCOra8LCoNKAwVyNXXsAjZ3vjd1tmsQMlhZI5Jq1xHarKktDD6IeBylOeUPxegISaqjIffeZZ3ABAEJdx6zbxJ+DK4u0rCTYhX+v7n8q+wWiFE2768tgpB1Qu/JMZjPt8iVeqRENoXiLrheR1ZVXReUstgk7qoA3JofBmSPAzrmS4tSMEBo3u/zwqQn68gUrMk/mTxUUmidxqZMidI1TRAoQvWpstRp4vpKQFv6PaqXMpqvMWXLWjZulRyW00yVXNo7IZhoLQxGcz8NkdiYnYDzf/sSobbOWjVWnmmk5uYcDX8rTuTXfRN19R1JxTmYXZPY1Vv6L90J1881Bkk9oWJn/3Eycw7uTii5QrDPmBGAsZJew34GIp/jySstE5eRQ4w0RHZiRxSR/fgI6/1e66S2bmXxK6X5RiV9EKdJM6uk0FwNHtUODFca/vrqWKQay6bJQ0S5SGsvgiell81Fz0pzWRRoc= ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T07:51:33.675 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuJvGIPV7FFdFaQZp6plHTpj3SLxFP3boKwhAWCOra8LCoNKAwVyNXXsAjZ3vjd1tmsQMlhZI5Jq1xHarKktDD6IeBylOeUPxegISaqjIffeZZ3ABAEJdx6zbxJ+DK4u0rCTYhX+v7n8q+wWiFE2768tgpB1Qu/JMZjPt8iVeqRENoXiLrheR1ZVXReUstgk7qoA3JofBmSPAzrmS4tSMEBo3u/zwqQn68gUrMk/mTxUUmidxqZMidI1TRAoQvWpstRp4vpKQFv6PaqXMpqvMWXLWjZulRyW00yVXNo7IZhoLQxGcz8NkdiYnYDzf/sSobbOWjVWnmmk5uYcDX8rTuTXfRN19R1JxTmYXZPY1Vv6L90J1881Bkk9oWJn/3Eycw7uTii5QrDPmBGAsZJew34GIp/jySstE5eRQ4w0RHZiRxSR/fgI6/1e66S2bmXxK6X5RiV9EKdJM6uk0FwNHtUODFca/vrqWKQay6bJQ0S5SGsvgiell81Fz0pzWRRoc= ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:51:33.691 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuJvGIPV7FFdFaQZp6plHTpj3SLxFP3boKwhAWCOra8LCoNKAwVyNXXsAjZ3vjd1tmsQMlhZI5Jq1xHarKktDD6IeBylOeUPxegISaqjIffeZZ3ABAEJdx6zbxJ+DK4u0rCTYhX+v7n8q+wWiFE2768tgpB1Qu/JMZjPt8iVeqRENoXiLrheR1ZVXReUstgk7qoA3JofBmSPAzrmS4tSMEBo3u/zwqQn68gUrMk/mTxUUmidxqZMidI1TRAoQvWpstRp4vpKQFv6PaqXMpqvMWXLWjZulRyW00yVXNo7IZhoLQxGcz8NkdiYnYDzf/sSobbOWjVWnmmk5uYcDX8rTuTXfRN19R1JxTmYXZPY1Vv6L90J1881Bkk9oWJn/3Eycw7uTii5QrDPmBGAsZJew34GIp/jySstE5eRQ4w0RHZiRxSR/fgI6/1e66S2bmXxK6X5RiV9EKdJM6uk0FwNHtUODFca/vrqWKQay6bJQ0S5SGsvgiell81Fz0pzWRRoc= ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T07:51:33.738 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuJvGIPV7FFdFaQZp6plHTpj3SLxFP3boKwhAWCOra8LCoNKAwVyNXXsAjZ3vjd1tmsQMlhZI5Jq1xHarKktDD6IeBylOeUPxegISaqjIffeZZ3ABAEJdx6zbxJ+DK4u0rCTYhX+v7n8q+wWiFE2768tgpB1Qu/JMZjPt8iVeqRENoXiLrheR1ZVXReUstgk7qoA3JofBmSPAzrmS4tSMEBo3u/zwqQn68gUrMk/mTxUUmidxqZMidI1TRAoQvWpstRp4vpKQFv6PaqXMpqvMWXLWjZulRyW00yVXNo7IZhoLQxGcz8NkdiYnYDzf/sSobbOWjVWnmmk5uYcDX8rTuTXfRN19R1JxTmYXZPY1Vv6L90J1881Bkk9oWJn/3Eycw7uTii5QrDPmBGAsZJew34GIp/jySstE5eRQ4w0RHZiRxSR/fgI6/1e66S2bmXxK6X5RiV9EKdJM6uk0FwNHtUODFca/vrqWKQay6bJQ0S5SGsvgiell81Fz0pzWRRoc= ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:51:33.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-20T07:51:33.904 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:33 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3169318372' entity='client.admin' 2024-02-20T07:51:34.010 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 -- 172.21.15.55:0/3290470071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 msgr2=0x7f88200fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 --2- 172.21.15.55:0/3290470071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f88200fcfe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f881c0048c0 tx=0x7f881c009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 -- 172.21.15.55:0/3290470071 shutdown_connections 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 --2- 172.21.15.55:0/3290470071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f88200fcfe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 -- 172.21.15.55:0/3290470071 >> 172.21.15.55:0/3290470071 conn(0x7f88200f8270 msgr2=0x7f88200fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 -- 172.21.15.55:0/3290470071 shutdown_connections 2024-02-20T07:51:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 -- 172.21.15.55:0/3290470071 wait complete. 2024-02-20T07:51:36.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.128+0000 7f8828406700 1 Processor -- start 2024-02-20T07:51:36.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8828406700 1 -- start start 2024-02-20T07:51:36.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8828406700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f882010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8828406700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882010d410 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f88261a2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f882010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f88261a2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f882010ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54870/0 (socket says 172.21.15.55:54870) 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f88261a2700 1 -- 172.21.15.55:0/319218236 learned_addr learned my addr 172.21.15.55:0/319218236 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f88261a2700 1 -- 172.21.15.55:0/319218236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c005040 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f88261a2700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f882010ced0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f881c009680 tx=0x7f881c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f881c00bed0 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f881c00b5c0 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882010d610 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f881c01c5b0 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.129+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882010e2b0 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.130+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f881c01c820 con 0x7f88200fcbd0 2024-02-20T07:51:36.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.130+0000 7f8816ffd700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f880c036820 0x7f880c038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:36.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.130+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882005f5c0 con 0x7f88200fcbd0 2024-02-20T07:51:36.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.130+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f881c049bc0 con 0x7f88200fcbd0 2024-02-20T07:51:36.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.130+0000 7f88259a1700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f880c036820 0x7f880c038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:36.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.131+0000 7f88259a1700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f880c036820 0x7f880c038cd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8810009990 tx=0x7f8810006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:36.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.135+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f881c011dd0 con 0x7f88200fcbd0 2024-02-20T07:51:36.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.335+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f882004bed0 con 0x7f88200fcbd0 2024-02-20T07:51:36.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.339+0000 7f8816ffd700 1 -- 172.21.15.55:0/319218236 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f881c011370 con 0x7f88200fcbd0 2024-02-20T07:51:36.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.341+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f880c036820 msgr2=0x7f880c038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:36.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.341+0000 7f8828406700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f880c036820 0x7f880c038cd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8810009990 tx=0x7f8810006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:36.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.341+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 msgr2=0x7f882010ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:36.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.341+0000 7f8828406700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f882010ced0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f881c009680 tx=0x7f881c009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:36.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.342+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 shutdown_connections 2024-02-20T07:51:36.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.342+0000 7f8828406700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f880c036820 0x7f880c038cd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.342+0000 7f8828406700 1 --2- 172.21.15.55:0/319218236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88200fcbd0 0x7f882010ced0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.342+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 >> 172.21.15.55:0/319218236 conn(0x7f88200f8270 msgr2=0x7f88200f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:36.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.342+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 shutdown_connections 2024-02-20T07:51:36.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:36.342+0000 7f8828406700 1 -- 172.21.15.55:0/319218236 wait complete. 2024-02-20T07:51:37.044 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-20T07:51:37.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-20T07:51:37.633 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:51:37.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:37 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/319218236' entity='client.admin' 2024-02-20T07:51:38.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:38.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:38.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:51:38.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:38.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:51:38.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T07:51:38.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:38.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:38 smithi055 ceph-mon[29254]: Deploying daemon crash.smithi055 on smithi055 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/3271979940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 msgr2=0x7fa0a40fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 --2- 172.21.15.55:0/3271979940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a40fbed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa0940048c0 tx=0x7fa094009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/3271979940 shutdown_connections 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 --2- 172.21.15.55:0/3271979940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a40fbed0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/3271979940 >> 172.21.15.55:0/3271979940 conn(0x7fa0a40f71a0 msgr2=0x7fa0a40f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/3271979940 shutdown_connections 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/3271979940 wait complete. 2024-02-20T07:51:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.060+0000 7fa0aa0a9700 1 Processor -- start 2024-02-20T07:51:40.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0aa0a9700 1 -- start start 2024-02-20T07:51:40.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0aa0a9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a410cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:40.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0aa0a9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a410d420 con 0x7fa0a40fbac0 2024-02-20T07:51:40.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0a37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a410cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:40.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0a37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a410cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54894/0 (socket says 172.21.15.55:54894) 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0a37fe700 1 -- 172.21.15.55:0/1625568247 learned_addr learned my addr 172.21.15.55:0/1625568247 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0a37fe700 1 -- 172.21.15.55:0/1625568247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa094005040 con 0x7fa0a40fbac0 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.061+0000 7fa0a37fe700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a410cee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa094009680 tx=0x7fa094009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.062+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa09400bed0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.062+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa09400b5c0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.062+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a410d620 con 0x7fa0a40fbac0 2024-02-20T07:51:40.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.062+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa09401c5b0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.062+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a410e2c0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.063+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fa09401c7d0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.063+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a405f5c0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.063+0000 7fa08bfff700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa08c036870 0x7fa08c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.063+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa094049a60 con 0x7fa0a40fbac0 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.063+0000 7fa0a2ffd700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa08c036870 0x7fa08c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:40.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.063+0000 7fa0a2ffd700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa08c036870 0x7fa08c038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa098009990 tx=0x7fa098006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:40.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.068+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa094011dd0 con 0x7fa0a40fbac0 2024-02-20T07:51:40.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.278+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a410eea0 con 0x7fa08c036870 2024-02-20T07:51:40.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.282+0000 7fa08bfff700 1 -- 172.21.15.55:0/1625568247 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa0a410eea0 con 0x7fa08c036870 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa08c036870 msgr2=0x7fa08c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa08c036870 0x7fa08c038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa098009990 tx=0x7fa098006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 msgr2=0x7fa0a410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a410cee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa094009680 tx=0x7fa094009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 shutdown_connections 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fa08c036870 0x7fa08c038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 --2- 172.21.15.55:0/1625568247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a40fbac0 0x7fa0a410cee0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:40.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.285+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 >> 172.21.15.55:0/1625568247 conn(0x7fa0a40f71a0 msgr2=0x7fa0a40661d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:40.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.286+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 shutdown_connections 2024-02-20T07:51:40.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:40.286+0000 7fa0aa0a9700 1 -- 172.21.15.55:0/1625568247 wait complete. 2024-02-20T07:51:40.834 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-02-20T07:51:40.834 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:51:40.834 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-02-20T07:51:40.852 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:51:40.853 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:51:40.911 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-02-20T07:51:40.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch host add smithi186 2024-02-20T07:51:41.356 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:51:41.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:41 smithi055 ceph-mon[29254]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:41.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:41 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:42.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:42 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:42.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:42 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:42.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:42 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:42.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:42 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:42.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:42 smithi055 ceph-mon[29254]: Deploying daemon node-exporter.smithi055 on smithi055 2024-02-20T07:51:42.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.897+0000 7f7ce3339700 1 -- 172.21.15.55:0/657948006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 msgr2=0x7f7cdc0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:42.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.897+0000 7f7ce3339700 1 --2- 172.21.15.55:0/657948006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc0fbed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80048c0 tx=0x7f7cd8009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:42.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 -- 172.21.15.55:0/657948006 shutdown_connections 2024-02-20T07:51:42.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 --2- 172.21.15.55:0/657948006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc0fbed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 -- 172.21.15.55:0/657948006 >> 172.21.15.55:0/657948006 conn(0x7f7cdc0f71a0 msgr2=0x7f7cdc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 -- 172.21.15.55:0/657948006 shutdown_connections 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 -- 172.21.15.55:0/657948006 wait complete. 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 Processor -- start 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 -- start start 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc10fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce3339700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0665f0 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.898+0000 7f7ce10d5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc10fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:42.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7ce10d5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc10fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33310/0 (socket says 172.21.15.55:33310) 2024-02-20T07:51:42.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7ce10d5700 1 -- 172.21.15.55:0/511832269 learned_addr learned my addr 172.21.15.55:0/511832269 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:42.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7ce10d5700 1 -- 172.21.15.55:0/511832269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd8005040 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7ce10d5700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc10fbe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009680 tx=0x7f7cd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd800bed0 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc0667f0 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cd800b5c0 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd801c5b0 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.899+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc110360 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.900+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f7cd801c820 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.900+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc105890 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.900+0000 7f7cd1ffb700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f7cc8036820 0x7f7cc8038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:42.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.901+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7cd8049ba0 con 0x7f7cdc0fbac0 2024-02-20T07:51:42.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.901+0000 7f7ce08d4700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f7cc8036820 0x7f7cc8038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:42.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.901+0000 7f7ce08d4700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f7cc8036820 0x7f7cc8038cd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009990 tx=0x7f7ccc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:42.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:42.905+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7cd8011550 con 0x7f7cdc0fbac0 2024-02-20T07:51:43.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:43.104+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cdc02b540 con 0x7f7cc8036820 2024-02-20T07:51:44.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:43 smithi055 ceph-mon[29254]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:44.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:43 smithi055 ceph-mon[29254]: Deploying cephadm binary to smithi186 2024-02-20T07:51:45.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.204+0000 7f7cd1ffb700 1 -- 172.21.15.55:0/511832269 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7cdc02b540 con 0x7f7cc8036820 2024-02-20T07:51:45.206 INFO:teuthology.orchestra.run.smithi055.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-02-20T07:51:45.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f7cc8036820 msgr2=0x7f7cc8038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:45.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f7cc8036820 0x7f7cc8038cd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009990 tx=0x7f7ccc006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:45.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 msgr2=0x7f7cdc10fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:45.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc10fbe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009680 tx=0x7f7cd8009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:45.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 shutdown_connections 2024-02-20T07:51:45.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f7cc8036820 0x7f7cc8038cd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:45.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 --2- 172.21.15.55:0/511832269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7cdc0fbac0 0x7f7cdc10fbe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:45.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 >> 172.21.15.55:0/511832269 conn(0x7f7cdc0f71a0 msgr2=0x7f7cdc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:45.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 shutdown_connections 2024-02-20T07:51:45.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:45.206+0000 7f7ce3339700 1 -- 172.21.15.55:0/511832269 wait complete. 2024-02-20T07:51:45.811 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:45 smithi055 ceph-mon[29254]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:45.811 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:45 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:45.811 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:45 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:45.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-20T07:51:46.097 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:51:46.968 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:46 smithi055 ceph-mon[29254]: Added host smithi186 2024-02-20T07:51:47.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:47 smithi055 ceph-mon[29254]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:47.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:47 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:47.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:47 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:47.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:47 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:47.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:47 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- 172.21.15.55:0/1094648194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 msgr2=0x7f4fc80fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 --2- 172.21.15.55:0/1094648194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc80fcfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00048c0 tx=0x7f4fc0009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- 172.21.15.55:0/1094648194 shutdown_connections 2024-02-20T07:51:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 --2- 172.21.15.55:0/1094648194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc80fcfe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- 172.21.15.55:0/1094648194 >> 172.21.15.55:0/1094648194 conn(0x7f4fc80f8270 msgr2=0x7f4fc80fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- 172.21.15.55:0/1094648194 shutdown_connections 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- 172.21.15.55:0/1094648194 wait complete. 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 Processor -- start 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- start start 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc8071e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.459+0000 7f4fccf11700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80702e0 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fc659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc8071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fc659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc8071e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33332/0 (socket says 172.21.15.55:33332) 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fc659c700 1 -- 172.21.15.55:0/3734703268 learned_addr learned my addr 172.21.15.55:0/3734703268 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fc659c700 1 -- 172.21.15.55:0/3734703268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0005040 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fc659c700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc8071e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009b80 tx=0x7f4fc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:48.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fc000b5c0 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fc000be70 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc8070480 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fc001c660 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.460+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc8070970 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.461+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f4fc0013420 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.461+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc81069a0 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.461+0000 7f4fb6ffd700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f4fb00368c0 0x7f4fb0038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:48.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.461+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4fc0049a10 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.461+0000 7f4fc5d9b700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f4fb00368c0 0x7f4fb0038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:48.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.462+0000 7f4fc5d9b700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f4fb00368c0 0x7f4fb0038d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009990 tx=0x7f4fbc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:48.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.466+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4fc0011550 con 0x7f4fc80fcbd0 2024-02-20T07:51:48.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.664+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4fc802b3b0 con 0x7f4fb00368c0 2024-02-20T07:51:48.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.665+0000 7f4fb6ffd700 1 -- 172.21.15.55:0/3734703268 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f4fc802b3b0 con 0x7f4fb00368c0 2024-02-20T07:51:48.667 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:51:48.667 INFO:teuthology.orchestra.run.smithi055.stdout:[{"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-02-20T07:51:48.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f4fb00368c0 msgr2=0x7f4fb0038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:48.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f4fb00368c0 0x7f4fb0038d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009990 tx=0x7f4fbc006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 msgr2=0x7f4fc8071e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc8071e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009b80 tx=0x7f4fc0009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 shutdown_connections 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f4fb00368c0 0x7f4fb0038d70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 --2- 172.21.15.55:0/3734703268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4fc80fcbd0 0x7f4fc8071e10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 >> 172.21.15.55:0/3734703268 conn(0x7f4fc80f8270 msgr2=0x7f4fc80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 shutdown_connections 2024-02-20T07:51:48.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:48.667+0000 7f4fccf11700 1 -- 172.21.15.55:0/3734703268 wait complete. 2024-02-20T07:51:48.949 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:48 smithi055 ceph-mon[29254]: Deploying daemon alertmanager.smithi055 on smithi055 2024-02-20T07:51:49.393 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-20T07:51:49.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-20T07:51:49.648 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:51:49.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:49 smithi055 ceph-mon[29254]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:49.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:49 smithi055 ceph-mon[29254]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:51:49.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:49 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:51.882 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:51 smithi055 ceph-mon[29254]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:51.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 -- 172.21.15.55:0/3732016775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 msgr2=0x7f06580fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:51.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 --2- 172.21.15.55:0/3732016775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f06580fcfe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06480048c0 tx=0x7f0648009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:51.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 -- 172.21.15.55:0/3732016775 shutdown_connections 2024-02-20T07:51:51.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 --2- 172.21.15.55:0/3732016775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f06580fcfe0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 -- 172.21.15.55:0/3732016775 >> 172.21.15.55:0/3732016775 conn(0x7f06580f8270 msgr2=0x7f06580fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 -- 172.21.15.55:0/3732016775 shutdown_connections 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 -- 172.21.15.55:0/3732016775 wait complete. 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.880+0000 7f065e266700 1 Processor -- start 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f065e266700 1 -- start start 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f065e266700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f065810f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f065e266700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065810f630 con 0x7f06580fcbd0 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f0657fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f065810f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f0657fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f065810f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33344/0 (socket says 172.21.15.55:33344) 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f0657fff700 1 -- 172.21.15.55:0/431890241 learned_addr learned my addr 172.21.15.55:0/431890241 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f0657fff700 1 -- 172.21.15.55:0/431890241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0648005040 con 0x7f06580fcbd0 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f0657fff700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f065810f0f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0648009b80 tx=0x7f0648009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:51.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.881+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f064800b5c0 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.882+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f064800be70 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.882+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065810f830 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.882+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f064801c660 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.882+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06581104d0 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.882+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f064801c7c0 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.883+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065805f5c0 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.883+0000 7f0654ff9700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f06400368c0 0x7f0640038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.883+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0648049ad0 con 0x7f06580fcbd0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.883+0000 7f06577fe700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f06400368c0 0x7f0640038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.883+0000 7f06577fe700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f06400368c0 0x7f0640038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f064c009990 tx=0x7f064c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:51.887+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0648011db0 con 0x7f06580fcbd0 2024-02-20T07:51:52.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.092+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f065805f5c0 con 0x7f06580fcbd0 2024-02-20T07:51:52.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.721+0000 7f0654ff9700 1 -- 172.21.15.55:0/431890241 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0648023990 con 0x7f06580fcbd0 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f06400368c0 msgr2=0x7f0640038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f06400368c0 0x7f0640038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f064c009990 tx=0x7f064c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 msgr2=0x7f065810f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f065810f0f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0648009b80 tx=0x7f0648009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 shutdown_connections 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f06400368c0 0x7f0640038d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 --2- 172.21.15.55:0/431890241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06580fcbd0 0x7f065810f0f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:52.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.723+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 >> 172.21.15.55:0/431890241 conn(0x7f06580f8270 msgr2=0x7f06580701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:52.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.724+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 shutdown_connections 2024-02-20T07:51:52.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:51:52.724+0000 7f065e266700 1 -- 172.21.15.55:0/431890241 wait complete. 2024-02-20T07:51:52.726 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2024-02-20T07:51:52.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:52 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/431890241' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-20T07:51:53.356 INFO:tasks.cephadm:Adding mon.smithi055 on smithi055 2024-02-20T07:51:53.356 INFO:tasks.cephadm:Adding mon.smithi186 on smithi186 2024-02-20T07:51:53.356 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi055:172.21.15.55=smithi055;smithi186:172.21.15.186=smithi186' 2024-02-20T07:51:53.604 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:51:53.651 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:51:54.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:53 smithi055 ceph-mon[29254]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:54.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:53 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/431890241' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-20T07:51:54.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:53 smithi055 ceph-mon[29254]: osdmap e4: 0 total, 0 up, 0 in 2024-02-20T07:51:55.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 -- 172.21.15.186:0/3678011242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 msgr2=0x7f09dc0fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:55.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 --2- 172.21.15.186:0/3678011242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc0fb290 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0048c0 tx=0x7f09cc009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 -- 172.21.15.186:0/3678011242 shutdown_connections 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 --2- 172.21.15.186:0/3678011242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc0fb290 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 -- 172.21.15.186:0/3678011242 >> 172.21.15.186:0/3678011242 conn(0x7f09dc0f6920 msgr2=0x7f09dc0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 -- 172.21.15.186:0/3678011242 shutdown_connections 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.156+0000 7f09e0ecd700 1 -- 172.21.15.186:0/3678011242 wait complete. 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09e0ecd700 1 Processor -- start 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09e0ecd700 1 -- start start 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09e0ecd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09e0ecd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc10d400 con 0x7f09dc0fae80 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09da59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09da59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:40702/0 (socket says 172.21.15.186:40702) 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.157+0000 7f09da59c700 1 -- 172.21.15.186:0/2082822528 learned_addr learned my addr 172.21.15.186:0/2082822528 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09da59c700 1 -- 172.21.15.186:0/2082822528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc005040 con 0x7f09dc0fae80 2024-02-20T07:51:55.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09da59c700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc10cec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f09cc000c00 tx=0x7f09cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:55.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc00b5c0 con 0x7f09dc0fae80 2024-02-20T07:51:55.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09cc00be70 con 0x7f09dc0fae80 2024-02-20T07:51:55.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc10d600 con 0x7f09dc0fae80 2024-02-20T07:51:55.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc01c590 con 0x7f09dc0fae80 2024-02-20T07:51:55.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.158+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc10e2a0 con 0x7f09dc0fae80 2024-02-20T07:51:55.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.159+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f09cc013420 con 0x7f09dc0fae80 2024-02-20T07:51:55.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.159+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc05f5c0 con 0x7f09dc0fae80 2024-02-20T07:51:55.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.159+0000 7f09caffd700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f09c4036870 0x7f09c4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:55.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.159+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f09cc049ab0 con 0x7f09dc0fae80 2024-02-20T07:51:55.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.159+0000 7f09d9d9b700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f09c4036870 0x7f09c4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:55.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.160+0000 7f09d9d9b700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f09c4036870 0x7f09c4038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009990 tx=0x7f09d0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:55.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.164+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f09cc013a80 con 0x7f09dc0fae80 2024-02-20T07:51:55.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.354+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 --> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi186:172.21.15.186=smithi186", "target": ["mon-mgr", ""]}) v1 -- 0x7f09dc10ef10 con 0x7f09c4036870 2024-02-20T07:51:55.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.363+0000 7f09caffd700 1 -- 172.21.15.186:0/2082822528 <== mgr.14160 v2:172.21.15.55:6800/2284801197 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f09dc10ef10 con 0x7f09c4036870 2024-02-20T07:51:55.364 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-02-20T07:51:55.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f09c4036870 msgr2=0x7f09c4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:55.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f09c4036870 0x7f09c4038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009990 tx=0x7f09d0006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:55.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 msgr2=0x7f09dc10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc10cec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f09cc000c00 tx=0x7f09cc009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 shutdown_connections 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f09c4036870 0x7f09c4038d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 --2- 172.21.15.186:0/2082822528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09dc0fae80 0x7f09dc10cec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 >> 172.21.15.186:0/2082822528 conn(0x7f09dc0f6920 msgr2=0x7f09dc0f7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 shutdown_connections 2024-02-20T07:51:55.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:55.365+0000 7f09e0ecd700 1 -- 172.21.15.186:0/2082822528 wait complete. 2024-02-20T07:51:56.079 DEBUG:teuthology.orchestra.run.smithi186:mon.smithi186> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi186.service 2024-02-20T07:51:56.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:51:56.082 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:51:56.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:55 smithi055 ceph-mon[29254]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:56.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:55 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:51:56.461 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:51:56.508 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:51:57.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:56 smithi055 ceph-mon[29254]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi186:172.21.15.186=smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:57.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:56 smithi055 ceph-mon[29254]: Saving service mon spec with placement smithi055:172.21.15.55=smithi055;smithi186:172.21.15.186=smithi186;count:2 2024-02-20T07:51:57.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.822+0000 7f678d80c700 1 -- 172.21.15.186:0/1377461199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 msgr2=0x7f67880fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.822+0000 7f678d80c700 1 --2- 172.21.15.186:0/1377461199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f67880fcfe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f67780048c0 tx=0x7f6778009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.822+0000 7f678d80c700 1 -- 172.21.15.186:0/1377461199 shutdown_connections 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.822+0000 7f678d80c700 1 --2- 172.21.15.186:0/1377461199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f67880fcfe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.822+0000 7f678d80c700 1 -- 172.21.15.186:0/1377461199 >> 172.21.15.186:0/1377461199 conn(0x7f67880f8270 msgr2=0x7f67880fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.823+0000 7f678d80c700 1 -- 172.21.15.186:0/1377461199 shutdown_connections 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.823+0000 7f678d80c700 1 -- 172.21.15.186:0/1377461199 wait complete. 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.823+0000 7f678d80c700 1 Processor -- start 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.823+0000 7f678d80c700 1 -- start start 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.823+0000 7f678d80c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f678810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.823+0000 7f678d80c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678810d410 con 0x7f67880fcbd0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f6786ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f678810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f6786ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f678810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:40730/0 (socket says 172.21.15.186:40730) 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f6786ffd700 1 -- 172.21.15.186:0/375687129 learned_addr learned my addr 172.21.15.186:0/375687129 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f6786ffd700 1 -- 172.21.15.186:0/375687129 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778005040 con 0x7f67880fcbd0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f6786ffd700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f678810ced0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6778009680 tx=0x7f6778009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f677800bed0 con 0x7f67880fcbd0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678810d610 con 0x7f67880fcbd0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.824+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f677800b5c0 con 0x7f67880fcbd0 2024-02-20T07:51:57.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.825+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f677801c5b0 con 0x7f67880fcbd0 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.825+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678810e2b0 con 0x7f67880fcbd0 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.825+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678805f5c0 con 0x7f67880fcbd0 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.825+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6778013a80 con 0x7f67880fcbd0 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.826+0000 7f676f7fe700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6770036820 0x7f6770038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.826+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6778049b70 con 0x7f67880fcbd0 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.826+0000 7f67867fc700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6770036820 0x7f6770038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:57.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.826+0000 7f67867fc700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6770036820 0x7f6770038cd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f677c009990 tx=0x7f677c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:57.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:57.831+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6778011940 con 0x7f67880fcbd0 2024-02-20T07:51:58.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.090+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f678810ee00 con 0x7f67880fcbd0 2024-02-20T07:51:58.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.091+0000 7f676f7fe700 1 -- 172.21.15.186:0/375687129 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6778011370 con 0x7f67880fcbd0 2024-02-20T07:51:58.092 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:51:58.093 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6770036820 msgr2=0x7f6770038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6770036820 0x7f6770038cd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f677c009990 tx=0x7f677c006e40 comp rx=0 tx=0).stop 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 msgr2=0x7f678810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f678810ced0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6778009680 tx=0x7f6778009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 shutdown_connections 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f6770036820 0x7f6770038cd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 --2- 172.21.15.186:0/375687129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67880fcbd0 0x7f678810ced0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 >> 172.21.15.186:0/375687129 conn(0x7f67880f8270 msgr2=0x7f67880f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 shutdown_connections 2024-02-20T07:51:58.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:51:58.093+0000 7f678d80c700 1 -- 172.21.15.186:0/375687129 wait complete. 2024-02-20T07:51:58.096 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:51:58.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:57 smithi055 ceph-mon[29254]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:59.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:58 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/375687129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:51:59.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:51:58 smithi055 ceph-mon[29254]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:59.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:51:59.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:51:59.887 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:51:59.934 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:01.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 -- 172.21.15.186:0/2178748146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 msgr2=0x7f27580fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:01.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 --2- 172.21.15.186:0/2178748146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f27580fbed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27500048c0 tx=0x7f2750009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:01.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 -- 172.21.15.186:0/2178748146 shutdown_connections 2024-02-20T07:52:01.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 --2- 172.21.15.186:0/2178748146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f27580fbed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:01.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 -- 172.21.15.186:0/2178748146 >> 172.21.15.186:0/2178748146 conn(0x7f27580f71a0 msgr2=0x7f27580f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:01.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 -- 172.21.15.186:0/2178748146 shutdown_connections 2024-02-20T07:52:01.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.304+0000 7f275d883700 1 -- 172.21.15.186:0/2178748146 wait complete. 2024-02-20T07:52:01.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f275d883700 1 Processor -- start 2024-02-20T07:52:01.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f275d883700 1 -- start start 2024-02-20T07:52:01.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f275d883700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f275810fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:01.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f275d883700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758066810 con 0x7f27580fbac0 2024-02-20T07:52:01.308 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f2756ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f275810fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f2756ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f275810fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:40752/0 (socket says 172.21.15.186:40752) 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f2756ffd700 1 -- 172.21.15.186:0/3234285319 learned_addr learned my addr 172.21.15.186:0/3234285319 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.305+0000 7f2756ffd700 1 -- 172.21.15.186:0/3234285319 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750005040 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.306+0000 7f2756ffd700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f275810fc00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2750009680 tx=0x7f2750009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.306+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f275000bed0 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.306+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2758066a10 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.306+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f275000b5c0 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.306+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f275001c5b0 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.306+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2758110380 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.307+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f2750013a80 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.307+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2758105890 con 0x7f27580fbac0 2024-02-20T07:52:01.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.307+0000 7f274f7fe700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f2744036820 0x7f2744038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:01.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.307+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2750049b70 con 0x7f27580fbac0 2024-02-20T07:52:01.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.307+0000 7f27567fc700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f2744036820 0x7f2744038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:01.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.308+0000 7f27567fc700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f2744036820 0x7f2744038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2740009990 tx=0x7f2740006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:01.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.312+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2750011940 con 0x7f27580fbac0 2024-02-20T07:52:01.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.568+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f275804bed0 con 0x7f27580fbac0 2024-02-20T07:52:01.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.568+0000 7f274f7fe700 1 -- 172.21.15.186:0/3234285319 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2750011370 con 0x7f27580fbac0 2024-02-20T07:52:01.570 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:01.570 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:01.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f2744036820 msgr2=0x7f2744038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:01.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f2744036820 0x7f2744038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2740009990 tx=0x7f2740006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:01.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 msgr2=0x7f275810fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:01.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f275810fc00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2750009680 tx=0x7f2750009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 shutdown_connections 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f2744036820 0x7f2744038cd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 --2- 172.21.15.186:0/3234285319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27580fbac0 0x7f275810fc00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 >> 172.21.15.186:0/3234285319 conn(0x7f27580f71a0 msgr2=0x7f27580f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 shutdown_connections 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:01.571+0000 7f275d883700 1 -- 172.21.15.186:0/3234285319 wait complete. 2024-02-20T07:52:01.573 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:01.775 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:01 smithi055 ceph-mon[29254]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:01.775 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:01 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3234285319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:03.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:03.073 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:03.324 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:03.370 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T07:52:03.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:03 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:04.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 -- 172.21.15.186:0/2986145035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 msgr2=0x7fdb080fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:04.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 --2- 172.21.15.186:0/2986145035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb080fbed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80048c0 tx=0x7fdaf8009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 -- 172.21.15.186:0/2986145035 shutdown_connections 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 --2- 172.21.15.186:0/2986145035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb080fbed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 -- 172.21.15.186:0/2986145035 >> 172.21.15.186:0/2986145035 conn(0x7fdb080f71a0 msgr2=0x7fdb080f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 -- 172.21.15.186:0/2986145035 shutdown_connections 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.853+0000 7fdb10099700 1 -- 172.21.15.186:0/2986145035 wait complete. 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb10099700 1 Processor -- start 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb10099700 1 -- start start 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb10099700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb0810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb10099700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0810d400 con 0x7fdb080fbac0 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb0de35700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb0810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb0de35700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb0810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:44164/0 (socket says 172.21.15.186:44164) 2024-02-20T07:52:04.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb0de35700 1 -- 172.21.15.186:0/3728505252 learned_addr learned my addr 172.21.15.186:0/3728505252 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.854+0000 7fdb0de35700 1 -- 172.21.15.186:0/3728505252 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf8005040 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.855+0000 7fdb0de35700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb0810cec0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009680 tx=0x7fdaf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.855+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaf800bed0 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.855+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaf800b5c0 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.855+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0810d600 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.855+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaf801c5b0 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.855+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0810e2a0 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.856+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fdaf8013420 con 0x7fdb080fbac0 2024-02-20T07:52:04.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.856+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0805f5c0 con 0x7fdb080fbac0 2024-02-20T07:52:04.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.856+0000 7fdafe7fc700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fdaf4036820 0x7fdaf4038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:04.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.856+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdaf8049b30 con 0x7fdb080fbac0 2024-02-20T07:52:04.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.856+0000 7fdb0d634700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fdaf4036820 0x7fdaf4038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:04.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.857+0000 7fdb0d634700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fdaf4036820 0x7fdaf4038cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009990 tx=0x7fdb04006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:04.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:04.861+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdaf8013a80 con 0x7fdb080fbac0 2024-02-20T07:52:04.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:04 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T07:52:04.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:04 smithi055 ceph-mon[29254]: Deploying daemon grafana.smithi055 on smithi055 2024-02-20T07:52:04.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:04 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:05.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.119+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb0810e640 con 0x7fdb080fbac0 2024-02-20T07:52:05.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.119+0000 7fdafe7fc700 1 -- 172.21.15.186:0/3728505252 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fdaf8011370 con 0x7fdb080fbac0 2024-02-20T07:52:05.121 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:05.121 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fdaf4036820 msgr2=0x7fdaf4038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fdaf4036820 0x7fdaf4038cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009990 tx=0x7fdb04006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 msgr2=0x7fdb0810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb0810cec0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009680 tx=0x7fdaf8009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 shutdown_connections 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fdaf4036820 0x7fdaf4038cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 --2- 172.21.15.186:0/3728505252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb080fbac0 0x7fdb0810cec0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 >> 172.21.15.186:0/3728505252 conn(0x7fdb080f71a0 msgr2=0x7fdb080f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 shutdown_connections 2024-02-20T07:52:05.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:05.121+0000 7fdb10099700 1 -- 172.21.15.186:0/3728505252 wait complete. 2024-02-20T07:52:05.124 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:05.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:05 smithi055 ceph-mon[29254]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:05.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:05 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3728505252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:06.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:06.686 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:06.928 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:06.974 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:07.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:07 smithi055 ceph-mon[29254]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:08.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 -- 172.21.15.186:0/1793348276 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 msgr2=0x7fc0580fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:08.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 --2- 172.21.15.186:0/1793348276 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc0580fcfe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc0480048c0 tx=0x7fc048009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 -- 172.21.15.186:0/1793348276 shutdown_connections 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 --2- 172.21.15.186:0/1793348276 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc0580fcfe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 -- 172.21.15.186:0/1793348276 >> 172.21.15.186:0/1793348276 conn(0x7fc0580f8270 msgr2=0x7fc0580fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 -- 172.21.15.186:0/1793348276 shutdown_connections 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.262+0000 7fc05e70b700 1 -- 172.21.15.186:0/1793348276 wait complete. 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc05e70b700 1 Processor -- start 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc05e70b700 1 -- start start 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc05e70b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc05810cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc05e70b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05810d4a0 con 0x7fc0580fcbd0 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc057fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc05810cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc057fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc05810cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:44182/0 (socket says 172.21.15.186:44182) 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.263+0000 7fc057fff700 1 -- 172.21.15.186:0/3351808674 learned_addr learned my addr 172.21.15.186:0/3351808674 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc057fff700 1 -- 172.21.15.186:0/3351808674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc048005040 con 0x7fc0580fcbd0 2024-02-20T07:52:08.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc057fff700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc05810cf60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc048009b80 tx=0x7fc048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:08.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04800b5c0 con 0x7fc0580fcbd0 2024-02-20T07:52:08.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05810d6a0 con 0x7fc0580fcbd0 2024-02-20T07:52:08.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc04800be70 con 0x7fc0580fcbd0 2024-02-20T07:52:08.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04801c660 con 0x7fc0580fcbd0 2024-02-20T07:52:08.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.264+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05810e2f0 con 0x7fc0580fcbd0 2024-02-20T07:52:08.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.265+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc048013420 con 0x7fc0580fcbd0 2024-02-20T07:52:08.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.265+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05805f5c0 con 0x7fc0580fcbd0 2024-02-20T07:52:08.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.265+0000 7fc054ff9700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fc040036870 0x7fc040038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:08.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.265+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc048049a80 con 0x7fc0580fcbd0 2024-02-20T07:52:08.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.265+0000 7fc0577fe700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fc040036870 0x7fc040038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:08.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.266+0000 7fc0577fe700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fc040036870 0x7fc040038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009990 tx=0x7fc04c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:08.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.274+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc048011550 con 0x7fc0580fcbd0 2024-02-20T07:52:08.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.529+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc05810ee40 con 0x7fc0580fcbd0 2024-02-20T07:52:08.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.529+0000 7fc054ff9700 1 -- 172.21.15.186:0/3351808674 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc048011370 con 0x7fc0580fcbd0 2024-02-20T07:52:08.531 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:08.531 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:08.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fc040036870 msgr2=0x7fc040038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:08.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fc040036870 0x7fc040038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009990 tx=0x7fc04c006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 msgr2=0x7fc05810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc05810cf60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc048009b80 tx=0x7fc048009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 shutdown_connections 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fc040036870 0x7fc040038d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 --2- 172.21.15.186:0/3351808674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0580fcbd0 0x7fc05810cf60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 >> 172.21.15.186:0/3351808674 conn(0x7fc0580f8270 msgr2=0x7fc0580f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 shutdown_connections 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:08.532+0000 7fc05e70b700 1 -- 172.21.15.186:0/3351808674 wait complete. 2024-02-20T07:52:08.534 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:09.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:08 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3351808674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:09.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:08 smithi055 ceph-mon[29254]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:10.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:10.045 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:10.290 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:10.337 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:11.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.708+0000 7f71c824f700 1 -- 172.21.15.186:0/2845693679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 msgr2=0x7f71c00fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.708+0000 7f71c824f700 1 --2- 172.21.15.186:0/2845693679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c00fcfe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0048c0 tx=0x7f71bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.708+0000 7f71c824f700 1 -- 172.21.15.186:0/2845693679 shutdown_connections 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.708+0000 7f71c824f700 1 --2- 172.21.15.186:0/2845693679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c00fcfe0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.708+0000 7f71c824f700 1 -- 172.21.15.186:0/2845693679 >> 172.21.15.186:0/2845693679 conn(0x7f71c00f8270 msgr2=0x7f71c00fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c824f700 1 -- 172.21.15.186:0/2845693679 shutdown_connections 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c824f700 1 -- 172.21.15.186:0/2845693679 wait complete. 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c824f700 1 Processor -- start 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c824f700 1 -- start start 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c824f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c010cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:11.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c824f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c010d4a0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c5feb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c010cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c5feb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c010cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:44204/0 (socket says 172.21.15.186:44204) 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.709+0000 7f71c5feb700 1 -- 172.21.15.186:0/4080062618 learned_addr learned my addr 172.21.15.186:0/4080062618 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71c5feb700 1 -- 172.21.15.186:0/4080062618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc005040 con 0x7f71c00fcbd0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71c5feb700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c010cf60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009b80 tx=0x7f71bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71bc00b5c0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c010d6a0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71bc00be70 con 0x7f71c00fcbd0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71bc01c660 con 0x7f71c00fcbd0 2024-02-20T07:52:11.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.710+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c010e2f0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.711+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f71bc01c8d0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.711+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c005f5c0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.711+0000 7f71b67fc700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f71ac036870 0x7f71ac038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:11.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.711+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f71bc049cd0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.711+0000 7f71c57ea700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f71ac036870 0x7f71ac038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:11.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.712+0000 7f71c57ea700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f71ac036870 0x7f71ac038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009990 tx=0x7f71b0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:11.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.716+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f71bc011db0 con 0x7f71c00fcbd0 2024-02-20T07:52:11.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:11 smithi055 ceph-mon[29254]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:11.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.971+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71c010ee40 con 0x7f71c00fcbd0 2024-02-20T07:52:11.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.972+0000 7f71b67fc700 1 -- 172.21.15.186:0/4080062618 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f71bc023990 con 0x7f71c00fcbd0 2024-02-20T07:52:11.973 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:11.973 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:11.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f71ac036870 msgr2=0x7f71ac038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:11.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f71ac036870 0x7f71ac038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009990 tx=0x7f71b0006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:11.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 msgr2=0x7f71c010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c010cf60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009b80 tx=0x7f71bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 shutdown_connections 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f71ac036870 0x7f71ac038d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 --2- 172.21.15.186:0/4080062618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c00fcbd0 0x7f71c010cf60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 >> 172.21.15.186:0/4080062618 conn(0x7f71c00f8270 msgr2=0x7f71c00f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 shutdown_connections 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:11.974+0000 7f71c824f700 1 -- 172.21.15.186:0/4080062618 wait complete. 2024-02-20T07:52:11.976 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:12 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/4080062618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:13.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:13.532 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:13.779 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:13.826 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:13.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:13 smithi055 ceph-mon[29254]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.135+0000 7f0520b89700 1 -- 172.21.15.186:0/3622638665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 msgr2=0x7f051c0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.135+0000 7f0520b89700 1 --2- 172.21.15.186:0/3622638665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c0fbed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f05100048c0 tx=0x7f0510009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 -- 172.21.15.186:0/3622638665 shutdown_connections 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 --2- 172.21.15.186:0/3622638665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c0fbed0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 -- 172.21.15.186:0/3622638665 >> 172.21.15.186:0/3622638665 conn(0x7f051c0f71a0 msgr2=0x7f051c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 -- 172.21.15.186:0/3622638665 shutdown_connections 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 -- 172.21.15.186:0/3622638665 wait complete. 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 Processor -- start 2024-02-20T07:52:15.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 -- start start 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f0520b89700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c10d3d0 con 0x7f051c0fbac0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f051a59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f051a59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:46866/0 (socket says 172.21.15.186:46866) 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.136+0000 7f051a59c700 1 -- 172.21.15.186:0/3657272459 learned_addr learned my addr 172.21.15.186:0/3657272459 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f051a59c700 1 -- 172.21.15.186:0/3657272459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510005040 con 0x7f051c0fbac0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f051a59c700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c10ce90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0510009b80 tx=0x7f0510009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051000b5c0 con 0x7f051c0fbac0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c10d5d0 con 0x7f051c0fbac0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c10e270 con 0x7f051c0fbac0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f051000be70 con 0x7f051c0fbac0 2024-02-20T07:52:15.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.137+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05100255c0 con 0x7f051c0fbac0 2024-02-20T07:52:15.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.138+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f051001c3f0 con 0x7f051c0fbac0 2024-02-20T07:52:15.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.138+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc0052e0 con 0x7f051c0fbac0 2024-02-20T07:52:15.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.138+0000 7f050affd700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0504036820 0x7f0504038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:15.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.138+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0510049ad0 con 0x7f051c0fbac0 2024-02-20T07:52:15.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.139+0000 7f0519d9b700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0504036820 0x7f0504038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:15.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.139+0000 7f0519d9b700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0504036820 0x7f0504038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f050c009990 tx=0x7f050c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:15.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.144+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f051001a2f0 con 0x7f051c0fbac0 2024-02-20T07:52:15.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.399+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04fc005590 con 0x7f051c0fbac0 2024-02-20T07:52:15.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.399+0000 7f050affd700 1 -- 172.21.15.186:0/3657272459 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f051001a2f0 con 0x7f051c0fbac0 2024-02-20T07:52:15.401 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:15.401 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0504036820 msgr2=0x7f0504038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0504036820 0x7f0504038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f050c009990 tx=0x7f050c006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 msgr2=0x7f051c10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c10ce90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0510009b80 tx=0x7f0510009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 shutdown_connections 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0504036820 0x7f0504038cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 --2- 172.21.15.186:0/3657272459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051c0fbac0 0x7f051c10ce90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 >> 172.21.15.186:0/3657272459 conn(0x7f051c0f71a0 msgr2=0x7f051c0661d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 shutdown_connections 2024-02-20T07:52:15.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:15.401+0000 7f0520b89700 1 -- 172.21.15.186:0/3657272459 wait complete. 2024-02-20T07:52:15.404 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:15.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:15 smithi055 ceph-mon[29254]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:15.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:15 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3657272459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:16.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:16.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:17.208 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:17.256 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:17.785 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:17.785 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:17.785 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:17.786 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:17.786 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:17.786 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:17.786 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:17 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:18.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 -- 172.21.15.186:0/3184291098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 msgr2=0x7f07580fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:18.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 --2- 172.21.15.186:0/3184291098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580fbee0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f074c0048c0 tx=0x7f074c009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 -- 172.21.15.186:0/3184291098 shutdown_connections 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 --2- 172.21.15.186:0/3184291098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580fbee0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 -- 172.21.15.186:0/3184291098 >> 172.21.15.186:0/3184291098 conn(0x7f07580f7190 msgr2=0x7f07580f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 -- 172.21.15.186:0/3184291098 shutdown_connections 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.615+0000 7f075e932700 1 -- 172.21.15.186:0/3184291098 wait complete. 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f075e932700 1 Processor -- start 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f075e932700 1 -- start start 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f075e932700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f075e932700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580f76d0 con 0x7f07580fbad0 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f0757fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f0757fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:46882/0 (socket says 172.21.15.186:46882) 2024-02-20T07:52:18.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f0757fff700 1 -- 172.21.15.186:0/1622225437 learned_addr learned my addr 172.21.15.186:0/1622225437 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.616+0000 7f0757fff700 1 -- 172.21.15.186:0/1622225437 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c005040 con 0x7f07580fbad0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.617+0000 7f0757fff700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580f9250 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f074c009680 tx=0x7f074c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.617+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f074c00bed0 con 0x7f07580fbad0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.617+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f074c00b5c0 con 0x7f07580fbad0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.617+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07580f78d0 con 0x7f07580fbad0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.617+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f074c01c5b0 con 0x7f07580fbad0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.617+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07580f7dc0 con 0x7f07580fbad0 2024-02-20T07:52:18.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.618+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f074c013420 con 0x7f07580fbad0 2024-02-20T07:52:18.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.618+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07581068b0 con 0x7f07580fbad0 2024-02-20T07:52:18.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.618+0000 7f0754ff9700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0740036820 0x7f0740038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:18.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.618+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f074c049b70 con 0x7f07580fbad0 2024-02-20T07:52:18.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.618+0000 7f07577fe700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0740036820 0x7f0740038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:18.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.618+0000 7f07577fe700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0740036820 0x7f0740038cd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0748009990 tx=0x7f0748006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:18.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.623+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f074c013a80 con 0x7f07580fbad0 2024-02-20T07:52:18.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.877+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f075805f5c0 con 0x7f07580fbad0 2024-02-20T07:52:18.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.878+0000 7f0754ff9700 1 -- 172.21.15.186:0/1622225437 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f074c011370 con 0x7f07580fbad0 2024-02-20T07:52:18.879 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:18.880 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:18.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0740036820 msgr2=0x7f0740038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:18.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0740036820 0x7f0740038cd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0748009990 tx=0x7f0748006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 msgr2=0x7f07580f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580f9250 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f074c009680 tx=0x7f074c009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 shutdown_connections 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f0740036820 0x7f0740038cd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 --2- 172.21.15.186:0/1622225437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07580fbad0 0x7f07580f9250 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 >> 172.21.15.186:0/1622225437 conn(0x7f07580f7190 msgr2=0x7f0758065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 shutdown_connections 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:18.880+0000 7f075e932700 1 -- 172.21.15.186:0/1622225437 wait complete. 2024-02-20T07:52:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:19.124 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:18 smithi055 ceph-mon[29254]: Deploying daemon prometheus.smithi055 on smithi055 2024-02-20T07:52:20.136 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:19 smithi055 ceph-mon[29254]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:20.136 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:19 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/1622225437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:20.136 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:19 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:20.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:20.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:20.740 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:20.787 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:22.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 -- 172.21.15.186:0/2633935755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 msgr2=0x7f10cc0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:22.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 --2- 172.21.15.186:0/2633935755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc0fbed0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f10c4009c60 tx=0x7f10c4004d10 comp rx=0 tx=0).stop 2024-02-20T07:52:22.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 -- 172.21.15.186:0/2633935755 shutdown_connections 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 --2- 172.21.15.186:0/2633935755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc0fbed0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 -- 172.21.15.186:0/2633935755 >> 172.21.15.186:0/2633935755 conn(0x7f10cc0f71a0 msgr2=0x7f10cc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 -- 172.21.15.186:0/2633935755 shutdown_connections 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.127+0000 7f10d4309700 1 -- 172.21.15.186:0/2633935755 wait complete. 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d4309700 1 Processor -- start 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d4309700 1 -- start start 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d4309700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc10be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d4309700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc10c380 con 0x7f10cc0fbac0 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d20a5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc10be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d20a5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc10be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:46898/0 (socket says 172.21.15.186:46898) 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d20a5700 1 -- 172.21.15.186:0/2111224518 learned_addr learned my addr 172.21.15.186:0/2111224518 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.128+0000 7f10d20a5700 1 -- 172.21.15.186:0/2111224518 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c4009940 con 0x7f10cc0fbac0 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.129+0000 7f10d20a5700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc10be40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10c4015040 tx=0x7f10c4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:22.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.129+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10c4007c80 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.129+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10c401d910 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.129+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10c40129f0 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.129+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc10c580 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.129+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc10d140 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.130+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f10c4019040 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.130+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc05f5c0 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.130+0000 7f10c2ffd700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f10b8036870 0x7f10b8038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.130+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f10c404a5c0 con 0x7f10cc0fbac0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.130+0000 7f10d18a4700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f10b8036870 0x7f10b8038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.130+0000 7f10d18a4700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f10b8036870 0x7f10b8038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009990 tx=0x7f10bc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:22.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.135+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f10c4024310 con 0x7f10cc0fbac0 2024-02-20T07:52:22.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:21 smithi055 ceph-mon[29254]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:22.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.389+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10cc10dd20 con 0x7f10cc0fbac0 2024-02-20T07:52:22.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.390+0000 7f10c2ffd700 1 -- 172.21.15.186:0/2111224518 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f10c4017070 con 0x7f10cc0fbac0 2024-02-20T07:52:22.392 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:22.392 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:22.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f10b8036870 msgr2=0x7f10b8038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:22.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f10b8036870 0x7f10b8038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009990 tx=0x7f10bc006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:22.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 msgr2=0x7f10cc10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc10be40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10c4015040 tx=0x7f10c4004d10 comp rx=0 tx=0).stop 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 shutdown_connections 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f10b8036870 0x7f10b8038d20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 --2- 172.21.15.186:0/2111224518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10cc0fbac0 0x7f10cc10be40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 >> 172.21.15.186:0/2111224518 conn(0x7f10cc0f71a0 msgr2=0x7f10cc0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 shutdown_connections 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:22.392+0000 7f10d4309700 1 -- 172.21.15.186:0/2111224518 wait complete. 2024-02-20T07:52:22.394 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:23.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:22 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/2111224518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:23.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:23.986 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:24.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:23 smithi055 ceph-mon[29254]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:24.234 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:24.281 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:25.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 -- 172.21.15.186:0/442222043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 msgr2=0x7fcf400fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:25.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 --2- 172.21.15.186:0/442222043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf400fbed0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcf300048c0 tx=0x7fcf30009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 -- 172.21.15.186:0/442222043 shutdown_connections 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 --2- 172.21.15.186:0/442222043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf400fbed0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 -- 172.21.15.186:0/442222043 >> 172.21.15.186:0/442222043 conn(0x7fcf400f71a0 msgr2=0x7fcf400f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 -- 172.21.15.186:0/442222043 shutdown_connections 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 -- 172.21.15.186:0/442222043 wait complete. 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 Processor -- start 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.777+0000 7fcf46cf5700 1 -- start start 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf46cf5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf4010fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf46cf5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf40066870 con 0x7fcf400fbac0 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf44a91700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf4010fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf44a91700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf4010fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:46206/0 (socket says 172.21.15.186:46206) 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf44a91700 1 -- 172.21.15.186:0/34241352 learned_addr learned my addr 172.21.15.186:0/34241352 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:25.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf44a91700 1 -- 172.21.15.186:0/34241352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30005040 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.778+0000 7fcf44a91700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf4010fba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009b80 tx=0x7fcf30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf3000b5c0 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf3000be70 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf40066a70 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf3001c660 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf40110320 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fcf3001c7c0 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.779+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf40105890 con 0x7fcf400fbac0 2024-02-20T07:52:25.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.780+0000 7fcf3d7fa700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fcf280368c0 0x7fcf28038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:25.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.780+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcf30049a60 con 0x7fcf400fbac0 2024-02-20T07:52:25.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.780+0000 7fcf3ffff700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fcf280368c0 0x7fcf28038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:25.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.780+0000 7fcf3ffff700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fcf280368c0 0x7fcf28038d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009990 tx=0x7fcf34006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:25.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:25.785+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcf30011db0 con 0x7fcf400fbac0 2024-02-20T07:52:26.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.040+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf4004bed0 con 0x7fcf400fbac0 2024-02-20T07:52:26.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.040+0000 7fcf3d7fa700 1 -- 172.21.15.186:0/34241352 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fcf30023990 con 0x7fcf400fbac0 2024-02-20T07:52:26.042 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:26.042 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fcf280368c0 msgr2=0x7fcf28038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fcf280368c0 0x7fcf28038d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009990 tx=0x7fcf34006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 msgr2=0x7fcf4010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf4010fba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009b80 tx=0x7fcf30009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 shutdown_connections 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fcf280368c0 0x7fcf28038d70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 --2- 172.21.15.186:0/34241352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf400fbac0 0x7fcf4010fba0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 >> 172.21.15.186:0/34241352 conn(0x7fcf400f71a0 msgr2=0x7fcf400f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 shutdown_connections 2024-02-20T07:52:26.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:26.042+0000 7fcf46cf5700 1 -- 172.21.15.186:0/34241352 wait complete. 2024-02-20T07:52:26.045 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:26.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:25 smithi055 ceph-mon[29254]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:27.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:26 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/34241352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:27.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:27.791 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:28.040 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:28.087 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:28.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:27 smithi055 ceph-mon[29254]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:29.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.435+0000 7feaaa34d700 1 -- 172.21.15.186:0/1615960570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 msgr2=0x7feaa40fe650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:29.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.435+0000 7feaaa34d700 1 --2- 172.21.15.186:0/1615960570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa40fe650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fea980048c0 tx=0x7fea98009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.435+0000 7feaaa34d700 1 -- 172.21.15.186:0/1615960570 shutdown_connections 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.435+0000 7feaaa34d700 1 --2- 172.21.15.186:0/1615960570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa40fe650 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.435+0000 7feaaa34d700 1 -- 172.21.15.186:0/1615960570 >> 172.21.15.186:0/1615960570 conn(0x7feaa40f6920 msgr2=0x7feaa40f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.436+0000 7feaaa34d700 1 -- 172.21.15.186:0/1615960570 shutdown_connections 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.436+0000 7feaaa34d700 1 -- 172.21.15.186:0/1615960570 wait complete. 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.436+0000 7feaaa34d700 1 Processor -- start 2024-02-20T07:52:29.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.436+0000 7feaaa34d700 1 -- start start 2024-02-20T07:52:29.439 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaaa34d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa410cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:29.439 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaaa34d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa410d4b0 con 0x7feaa40fe240 2024-02-20T07:52:29.439 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa410cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa410cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:46230/0 (socket says 172.21.15.186:46230) 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa3fff700 1 -- 172.21.15.186:0/3517812079 learned_addr learned my addr 172.21.15.186:0/3517812079 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa3fff700 1 -- 172.21.15.186:0/3517812079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98005040 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa3fff700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa410cf70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fea98009f80 tx=0x7fea98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98004e60 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea98009650 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa410d6b0 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.437+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea9801c5b0 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.438+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa410e350 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.438+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fea98013980 con 0x7feaa40fe240 2024-02-20T07:52:29.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.438+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa405f5c0 con 0x7feaa40fe240 2024-02-20T07:52:29.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.439+0000 7feaa0ff9700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fea8c036870 0x7fea8c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:29.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.439+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fea98049540 con 0x7feaa40fe240 2024-02-20T07:52:29.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.439+0000 7feaa37fe700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fea8c036870 0x7fea8c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:29.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.439+0000 7feaa37fe700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fea8c036870 0x7fea8c038d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea94009990 tx=0x7fea94006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:29.445 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.444+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fea980137a0 con 0x7feaa40fe240 2024-02-20T07:52:29.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:28 smithi055 ceph-mon[29254]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:29.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.698+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feaa410eea0 con 0x7feaa40fe240 2024-02-20T07:52:29.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.698+0000 7feaa0ff9700 1 -- 172.21.15.186:0/3517812079 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fea98021030 con 0x7feaa40fe240 2024-02-20T07:52:29.700 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:29.700 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.700+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fea8c036870 msgr2=0x7fea8c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.700+0000 7feaaa34d700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fea8c036870 0x7fea8c038d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea94009990 tx=0x7fea94006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.700+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 msgr2=0x7feaa410cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.700+0000 7feaaa34d700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa410cf70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fea98009f80 tx=0x7fea98009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.701+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 shutdown_connections 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.701+0000 7feaaa34d700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fea8c036870 0x7fea8c038d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:29.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.701+0000 7feaaa34d700 1 --2- 172.21.15.186:0/3517812079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40fe240 0x7feaa410cf70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:29.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.701+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 >> 172.21.15.186:0/3517812079 conn(0x7feaa40f6920 msgr2=0x7feaa40f7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:29.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.701+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 shutdown_connections 2024-02-20T07:52:29.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:29.701+0000 7feaaa34d700 1 -- 172.21.15.186:0/3517812079 wait complete. 2024-02-20T07:52:29.703 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:30.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:29 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3517812079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:31.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:30 smithi055 ceph-mon[29254]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:31.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:31.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:31.487 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:31.535 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:33.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 -- 172.21.15.186:0/2445684098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 msgr2=0x7f5810101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:33.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 --2- 172.21.15.186:0/2445684098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f5810101ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58000048c0 tx=0x7f5800009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 -- 172.21.15.186:0/2445684098 shutdown_connections 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 --2- 172.21.15.186:0/2445684098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f5810101ba0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 -- 172.21.15.186:0/2445684098 >> 172.21.15.186:0/2445684098 conn(0x7f58100f7190 msgr2=0x7f58100f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 -- 172.21.15.186:0/2445684098 shutdown_connections 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.011+0000 7f5815632700 1 -- 172.21.15.186:0/2445684098 wait complete. 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.012+0000 7f5815632700 1 Processor -- start 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.012+0000 7f5815632700 1 -- start start 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.012+0000 7f5815632700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f581010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:33.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.012+0000 7f5815632700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581010f5c0 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f580effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f581010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f580effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f581010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:46236/0 (socket says 172.21.15.186:46236) 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f580effd700 1 -- 172.21.15.186:0/2997725579 learned_addr learned my addr 172.21.15.186:0/2997725579 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f580effd700 1 -- 172.21.15.186:0/2997725579 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5800005040 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f580effd700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f581010f080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5800009680 tx=0x7f5800009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580000b5c0 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581010f7c0 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5800004e60 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580001c590 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.013+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5810110460 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.014+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f5800013420 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.014+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581005f5c0 con 0x7f5810065dd0 2024-02-20T07:52:33.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.014+0000 7f58077fe700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f57fc036870 0x7f57fc038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:33.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.014+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5800049bb0 con 0x7f5810065dd0 2024-02-20T07:52:33.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.015+0000 7f580e7fc700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f57fc036870 0x7f57fc038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:33.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.015+0000 7f580e7fc700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f57fc036870 0x7f57fc038d20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009990 tx=0x7f57f8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:33.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.019+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5800025400 con 0x7f5810065dd0 2024-02-20T07:52:33.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.275+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5810110fb0 con 0x7f5810065dd0 2024-02-20T07:52:33.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.275+0000 7f58077fe700 1 -- 172.21.15.186:0/2997725579 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5800011370 con 0x7f5810065dd0 2024-02-20T07:52:33.278 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:33.278 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f57fc036870 msgr2=0x7f57fc038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f57fc036870 0x7f57fc038d20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009990 tx=0x7f57f8006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 msgr2=0x7f581010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f581010f080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5800009680 tx=0x7f5800009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 shutdown_connections 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f57fc036870 0x7f57fc038d20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 --2- 172.21.15.186:0/2997725579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5810065dd0 0x7f581010f080 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 >> 172.21.15.186:0/2997725579 conn(0x7f58100f7190 msgr2=0x7f58100f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 shutdown_connections 2024-02-20T07:52:33.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:33.277+0000 7f5815632700 1 -- 172.21.15.186:0/2997725579 wait complete. 2024-02-20T07:52:33.281 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:33.895 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:33 smithi055 ceph-mon[29254]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:33.895 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:33 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/2997725579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:34.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:34.838 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:35.079 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:35.125 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:35.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:35 smithi055 ceph-mon[29254]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:52:35.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:35 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:35.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:35 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:35.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:35 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' 2024-02-20T07:52:35.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:35 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 -- 172.21.15.186:0/2846834294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 msgr2=0x7fed9c101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 --2- 172.21.15.186:0/2846834294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c101b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fed94009c60 tx=0x7fed94004d10 comp rx=0 tx=0).stop 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 -- 172.21.15.186:0/2846834294 shutdown_connections 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 --2- 172.21.15.186:0/2846834294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c101b70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 -- 172.21.15.186:0/2846834294 >> 172.21.15.186:0/2846834294 conn(0x7fed9c0f71a0 msgr2=0x7fed9c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 -- 172.21.15.186:0/2846834294 shutdown_connections 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.632+0000 7feda304e700 1 -- 172.21.15.186:0/2846834294 wait complete. 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.633+0000 7feda304e700 1 Processor -- start 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.633+0000 7feda304e700 1 -- start start 2024-02-20T07:52:36.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.633+0000 7feda304e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c10f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.633+0000 7feda304e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c10f610 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda0dea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c10f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda0dea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c10f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:41302/0 (socket says 172.21.15.186:41302) 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda0dea700 1 -- 172.21.15.186:0/3622641670 learned_addr learned my addr 172.21.15.186:0/3622641670 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda0dea700 1 -- 172.21.15.186:0/3622641670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94009940 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda0dea700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c10f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed94004e60 tx=0x7fed94004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed9401d440 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c10f810 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed9401da70 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed94012aa0 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.634+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c1104b0 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.635+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7fed94019040 con 0x7fed9c066880 2024-02-20T07:52:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.635+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c05f5c0 con 0x7fed9c066880 2024-02-20T07:52:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.635+0000 7fed997fa700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 0x7fed84038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.635+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fed9404a370 con 0x7fed9c066880 2024-02-20T07:52:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.635+0000 7fed9bfff700 1 -- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 msgr2=0x7fed84038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2284801197 2024-02-20T07:52:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.635+0000 7fed9bfff700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 0x7fed84038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:52:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.640+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fed94012410 con 0x7fed9c066880 2024-02-20T07:52:36.838 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.836+0000 7fed9bfff700 1 -- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 msgr2=0x7fed84038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2284801197 2024-02-20T07:52:36.838 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.836+0000 7fed9bfff700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 0x7fed84038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:52:36.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.897+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed9c110fb0 con 0x7fed9c066880 2024-02-20T07:52:36.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.898+0000 7fed997fa700 1 -- 172.21.15.186:0/3622641670 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fed94017020 con 0x7fed9c066880 2024-02-20T07:52:36.900 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:36.900 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:36.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 msgr2=0x7fed84038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:52:36.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 0x7fed84038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:36.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 msgr2=0x7fed9c10f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:36.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c10f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed94004e60 tx=0x7fed94004d10 comp rx=0 tx=0).stop 2024-02-20T07:52:36.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 shutdown_connections 2024-02-20T07:52:36.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7fed840368c0 0x7fed84038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:36.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 --2- 172.21.15.186:0/3622641670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed9c066880 0x7fed9c10f0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:36.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 >> 172.21.15.186:0/3622641670 conn(0x7fed9c0f71a0 msgr2=0x7fed9c0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:36.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 shutdown_connections 2024-02-20T07:52:36.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:36.900+0000 7feda304e700 1 -- 172.21.15.186:0/3622641670 wait complete. 2024-02-20T07:52:36.903 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:37.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:37 smithi055 ceph-mon[29254]: from='mgr.14160 172.21.15.55:0/493047285' entity='mgr.smithi055.stjper' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-20T07:52:37.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:37 smithi055 ceph-mon[29254]: mgrmap e14: smithi055.stjper(active, since 71s) 2024-02-20T07:52:37.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:37 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3622641670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:38.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:38.495 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:38.747 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:38.793 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:40.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 -- 172.21.15.186:0/3271817527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 msgr2=0x7f59e00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:40.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 --2- 172.21.15.186:0/3271817527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e00fd4e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009c60 tx=0x7f59d8004d10 comp rx=0 tx=0).stop 2024-02-20T07:52:40.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 -- 172.21.15.186:0/3271817527 shutdown_connections 2024-02-20T07:52:40.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 --2- 172.21.15.186:0/3271817527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e00fd4e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:40.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 -- 172.21.15.186:0/3271817527 >> 172.21.15.186:0/3271817527 conn(0x7f59e00f6920 msgr2=0x7f59e00f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:40.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 -- 172.21.15.186:0/3271817527 shutdown_connections 2024-02-20T07:52:40.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.097+0000 7f59e5ba6700 1 -- 172.21.15.186:0/3271817527 wait complete. 2024-02-20T07:52:40.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59e5ba6700 1 Processor -- start 2024-02-20T07:52:40.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59e5ba6700 1 -- start start 2024-02-20T07:52:40.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59e5ba6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:40.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59e5ba6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e010d420 con 0x7f59e00fd0d0 2024-02-20T07:52:40.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59df7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:40.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59df7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e010cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:41318/0 (socket says 172.21.15.186:41318) 2024-02-20T07:52:40.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59df7fe700 1 -- 172.21.15.186:0/4111072864 learned_addr learned my addr 172.21.15.186:0/4111072864 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:40.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.098+0000 7f59df7fe700 1 -- 172.21.15.186:0/4111072864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8009940 con 0x7f59e00fd0d0 2024-02-20T07:52:40.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.099+0000 7f59df7fe700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e010cee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f59d8015040 tx=0x7f59d8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:40.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.099+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d8007c80 con 0x7f59e00fd0d0 2024-02-20T07:52:40.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.099+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e010d620 con 0x7f59e00fd0d0 2024-02-20T07:52:40.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.099+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d801d910 con 0x7f59e00fd0d0 2024-02-20T07:52:40.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.099+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d80129f0 con 0x7f59e00fd0d0 2024-02-20T07:52:40.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.099+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e010e2c0 con 0x7f59e00fd0d0 2024-02-20T07:52:40.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.100+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f59d8019040 con 0x7f59e00fd0d0 2024-02-20T07:52:40.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.100+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e005f5c0 con 0x7f59e00fd0d0 2024-02-20T07:52:40.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.100+0000 7f59c7fff700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 0x7f59c8038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:40.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.100+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f59d804a520 con 0x7f59e00fd0d0 2024-02-20T07:52:40.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.100+0000 7f59deffd700 1 -- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 msgr2=0x7f59c8038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2284801197 2024-02-20T07:52:40.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.100+0000 7f59deffd700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 0x7f59c8038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:52:40.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.109+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f59d802eaf0 con 0x7f59e00fd0d0 2024-02-20T07:52:40.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.301+0000 7f59deffd700 1 -- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 msgr2=0x7f59c8038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2284801197 2024-02-20T07:52:40.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.301+0000 7f59deffd700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 0x7f59c8038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:52:40.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.364+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59e010e5e0 con 0x7f59e00fd0d0 2024-02-20T07:52:40.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.365+0000 7f59c7fff700 1 -- 172.21.15.186:0/4111072864 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f59d8017070 con 0x7f59e00fd0d0 2024-02-20T07:52:40.366 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:40.366 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:40.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 msgr2=0x7f59c8038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:52:40.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 0x7f59c8038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:40.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 msgr2=0x7f59e010cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:40.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e010cee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f59d8015040 tx=0x7f59d8004d10 comp rx=0 tx=0).stop 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 shutdown_connections 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:6800/2284801197,v1:172.21.15.55:6801/2284801197] conn(0x7f59c80368c0 0x7f59c8038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 --2- 172.21.15.186:0/4111072864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59e00fd0d0 0x7f59e010cee0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 >> 172.21.15.186:0/4111072864 conn(0x7f59e00f6920 msgr2=0x7f59e00f7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 shutdown_connections 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:40.367+0000 7f59e5ba6700 1 -- 172.21.15.186:0/4111072864 wait complete. 2024-02-20T07:52:40.369 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:40.692 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:40 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/4111072864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:41.924 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:41.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:42.174 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:42.221 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:43.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.663+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/787214314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 msgr2=0x7f3fac0fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.663+0000 7f3fb0eb0700 1 --2- 172.21.15.186:0/787214314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac0fbee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00048c0 tx=0x7f3fa0009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.663+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/787214314 shutdown_connections 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.663+0000 7f3fb0eb0700 1 --2- 172.21.15.186:0/787214314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac0fbee0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.663+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/787214314 >> 172.21.15.186:0/787214314 conn(0x7f3fac0f7190 msgr2=0x7f3fac0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.664+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/787214314 shutdown_connections 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.664+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/787214314 wait complete. 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.664+0000 7f3fb0eb0700 1 Processor -- start 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.664+0000 7f3fb0eb0700 1 -- start start 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.664+0000 7f3fb0eb0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.664+0000 7f3fb0eb0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac10d400 con 0x7f3fac0fbad0 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3faa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:43.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3faa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:55210/0 (socket says 172.21.15.186:55210) 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3faa59c700 1 -- 172.21.15.186:0/2361579371 learned_addr learned my addr 172.21.15.186:0/2361579371 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3faa59c700 1 -- 172.21.15.186:0/2361579371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa0005040 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3faa59c700 1 --2- 172.21.15.186:0/2361579371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac10cec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009680 tx=0x7f3fa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fa000bed0 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fa000b5c0 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac10d600 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fa001c5b0 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.665+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac10e2a0 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.666+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac05f5c0 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.666+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7f3fa001c820 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.666+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3fa0049960 con 0x7f3fac0fbad0 2024-02-20T07:52:43.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.671+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3fa0011550 con 0x7f3fac0fbad0 2024-02-20T07:52:43.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.943+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fac10e620 con 0x7f3fac0fbad0 2024-02-20T07:52:43.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.944+0000 7f3f9affd700 1 -- 172.21.15.186:0/2361579371 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3fa0004d30 con 0x7f3fac0fbad0 2024-02-20T07:52:43.945 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:43.946 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:43.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 msgr2=0x7f3fac10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 --2- 172.21.15.186:0/2361579371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac10cec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009680 tx=0x7f3fa0009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 shutdown_connections 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 --2- 172.21.15.186:0/2361579371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fac0fbad0 0x7f3fac10cec0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 >> 172.21.15.186:0/2361579371 conn(0x7f3fac0f7190 msgr2=0x7f3fac0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 shutdown_connections 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:43.946+0000 7f3fb0eb0700 1 -- 172.21.15.186:0/2361579371 wait complete. 2024-02-20T07:52:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: Active manager daemon smithi055.stjper restarted 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: Activating manager daemon smithi055.stjper 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: osdmap e5: 0 total, 0 up, 0 in 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: mgrmap e15: smithi055.stjper(active, starting, since 0.00758913s) 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr metadata", "who": "smithi055.stjper", "id": "smithi055.stjper"}]: dispatch 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: Manager daemon smithi055.stjper is now available 2024-02-20T07:52:44.258 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.stjper/trash_purge_schedule"}]: dispatch 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/2361579371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:44.259 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:43 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:45.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:45.552 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:45.559 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:45 smithi055 ceph-mon[29254]: [20/Feb/2024:07:52:43] ENGINE Bus STARTING 2024-02-20T07:52:45.559 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:45 smithi055 ceph-mon[29254]: [20/Feb/2024:07:52:43] ENGINE Serving on https://172.21.15.55:7150 2024-02-20T07:52:45.559 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:45 smithi055 ceph-mon[29254]: [20/Feb/2024:07:52:43] ENGINE Bus STARTED 2024-02-20T07:52:45.559 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:45 smithi055 ceph-mon[29254]: mgrmap e16: smithi055.stjper(active, since 1.00971s) 2024-02-20T07:52:45.559 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:46.462 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:46.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:46.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:46 smithi055 ceph-mon[29254]: mgrmap e17: smithi055.stjper(active, since 3s) 2024-02-20T07:52:46.524 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:48.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:48.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:48.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 -- 172.21.15.186:0/2261279131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 msgr2=0x7f84e40fe020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 --2- 172.21.15.186:0/2261279131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e40fe020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f84e00048c0 tx=0x7f84e0009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 -- 172.21.15.186:0/2261279131 shutdown_connections 2024-02-20T07:52:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 --2- 172.21.15.186:0/2261279131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e40fe020 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 -- 172.21.15.186:0/2261279131 >> 172.21.15.186:0/2261279131 conn(0x7f84e4072850 msgr2=0x7f84e4072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 -- 172.21.15.186:0/2261279131 shutdown_connections 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 -- 172.21.15.186:0/2261279131 wait complete. 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 Processor -- start 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.268+0000 7f84eba0e700 1 -- start start 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84eba0e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e4112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84eba0e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4113500 con 0x7f84e40fdc50 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84e97aa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e4112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84e97aa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e4112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:55226/0 (socket says 172.21.15.186:55226) 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84e97aa700 1 -- 172.21.15.186:0/2282608674 learned_addr learned my addr 172.21.15.186:0/2282608674 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84e97aa700 1 -- 172.21.15.186:0/2282608674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e0005040 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84e97aa700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e4112fc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009330 tx=0x7f84e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84e000be50 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84e0009630 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e4113700 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.269+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84e001c3f0 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.270+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e4113b70 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.271+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f84e001c6b0 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.271+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e405f5c0 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.271+0000 7f84d9ffb700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f84d0036960 0x7f84d0038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.271+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f84e0049e20 con 0x7f84e40fdc50 2024-02-20T07:52:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.271+0000 7f84e8fa9700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f84d0036960 0x7f84d0038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:49.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.272+0000 7f84e8fa9700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f84d0036960 0x7f84d0038e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009990 tx=0x7f84d4006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.276+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f84e0023300 con 0x7f84e40fdc50 2024-02-20T07:52:49.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.555+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84e410e5d0 con 0x7f84e40fdc50 2024-02-20T07:52:49.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.556+0000 7f84d9ffb700 1 -- 172.21.15.186:0/2282608674 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f84e0011b60 con 0x7f84e40fdc50 2024-02-20T07:52:49.557 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:49.557 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f84d0036960 msgr2=0x7f84d0038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f84d0036960 0x7f84d0038e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009990 tx=0x7f84d4006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 msgr2=0x7f84e4112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e4112fc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009330 tx=0x7f84e0009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 shutdown_connections 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f84d0036960 0x7f84d0038e10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 --2- 172.21.15.186:0/2282608674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e40fdc50 0x7f84e4112fc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 >> 172.21.15.186:0/2282608674 conn(0x7f84e4072850 msgr2=0x7f84e4101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 shutdown_connections 2024-02-20T07:52:49.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:49.558+0000 7f84eba0e700 1 -- 172.21.15.186:0/2282608674 wait complete. 2024-02-20T07:52:49.561 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: Updating smithi055:/etc/ceph/ceph.conf 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: Updating smithi186:/etc/ceph/ceph.conf 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: Updating smithi055:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: Updating smithi186:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:49.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi186", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:52:49.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi186", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T07:52:49.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:50.636 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:50 smithi055 ceph-mon[29254]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:52:50.637 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:50 smithi055 ceph-mon[29254]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:52:50.637 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:50 smithi055 ceph-mon[29254]: Updating smithi055:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-20T07:52:50.637 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:50 smithi055 ceph-mon[29254]: Updating smithi186:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-20T07:52:50.637 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:50 smithi055 ceph-mon[29254]: Deploying daemon crash.smithi186 on smithi186 2024-02-20T07:52:50.637 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:50 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/2282608674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:51.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:51.172 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:51.432 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:52:53.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.203+0000 7f0cf7b35700 1 -- 172.21.15.186:0/2154988478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 msgr2=0x7f0cf00fb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:53.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.203+0000 7f0cf7b35700 1 --2- 172.21.15.186:0/2154988478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf00fb770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80048c0 tx=0x7f0ce8009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:53.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.203+0000 7f0cf7b35700 1 -- 172.21.15.186:0/2154988478 shutdown_connections 2024-02-20T07:52:53.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.203+0000 7f0cf7b35700 1 --2- 172.21.15.186:0/2154988478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf00fb770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.203+0000 7f0cf7b35700 1 -- 172.21.15.186:0/2154988478 >> 172.21.15.186:0/2154988478 conn(0x7f0cf00720b0 msgr2=0x7f0cf00724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.204+0000 7f0cf7b35700 1 -- 172.21.15.186:0/2154988478 shutdown_connections 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.204+0000 7f0cf7b35700 1 -- 172.21.15.186:0/2154988478 wait complete. 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.204+0000 7f0cf7b35700 1 Processor -- start 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.204+0000 7f0cf7b35700 1 -- start start 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.204+0000 7f0cf7b35700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf0112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.204+0000 7f0cf7b35700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0113530 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf58d1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf0112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf58d1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf0112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:55256/0 (socket says 172.21.15.186:55256) 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf58d1700 1 -- 172.21.15.186:0/3580498983 learned_addr learned my addr 172.21.15.186:0/3580498983 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf58d1700 1 -- 172.21.15.186:0/3580498983 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8005040 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf58d1700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf0112ff0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009300 tx=0x7f0ce8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce8004e60 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf0113730 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ce8013420 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf0113ba0 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.205+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce8025560 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.206+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f0ce8013a90 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.206+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf005f5c0 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.206+0000 7f0cee7fc700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0cd8036960 0x7f0cd8038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.206+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0ce8049b30 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.206+0000 7f0cf50d0700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0cd8036960 0x7f0cd8038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:53.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.207+0000 7f0cf50d0700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0cd8036960 0x7f0cd8038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009990 tx=0x7f0ce4006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:53.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.211+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0ce8023300 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.492+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cf0106f10 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.493+0000 7f0cee7fc700 1 -- 172.21.15.186:0/3580498983 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0ce8011b60 con 0x7f0cf00fb3a0 2024-02-20T07:52:53.495 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:53.495 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:53.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0cd8036960 msgr2=0x7f0cd8038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:53.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0cd8036960 0x7f0cd8038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009990 tx=0x7f0ce4006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:53.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 msgr2=0x7f0cf0112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:53.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf0112ff0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009300 tx=0x7f0ce8009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:53.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 shutdown_connections 2024-02-20T07:52:53.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0cd8036960 0x7f0cd8038e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:53.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 --2- 172.21.15.186:0/3580498983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf00fb3a0 0x7f0cf0112ff0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:53.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.495+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 >> 172.21.15.186:0/3580498983 conn(0x7f0cf00720b0 msgr2=0x7f0cf01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:53.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.496+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 shutdown_connections 2024-02-20T07:52:53.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:53.496+0000 7f0cf7b35700 1 -- 172.21.15.186:0/3580498983 wait complete. 2024-02-20T07:52:53.498 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:54.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:54 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3580498983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:52:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:54 smithi055 ceph-mon[29254]: Deploying daemon node-exporter.smithi186 on smithi186 2024-02-20T07:52:55.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:52:55.019 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:52:55.277 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.225+0000 7fb592402700 1 -- 172.21.15.186:0/224330859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 msgr2=0x7fb58c101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.225+0000 7fb592402700 1 --2- 172.21.15.186:0/224330859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c101f10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0048c0 tx=0x7fb57c009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.225+0000 7fb592402700 1 -- 172.21.15.186:0/224330859 shutdown_connections 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.225+0000 7fb592402700 1 --2- 172.21.15.186:0/224330859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c101f10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.225+0000 7fb592402700 1 -- 172.21.15.186:0/224330859 >> 172.21.15.186:0/224330859 conn(0x7fb58c0f7660 msgr2=0x7fb58c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.226+0000 7fb592402700 1 -- 172.21.15.186:0/224330859 shutdown_connections 2024-02-20T07:52:58.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.226+0000 7fb592402700 1 -- 172.21.15.186:0/224330859 wait complete. 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.226+0000 7fb592402700 1 Processor -- start 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.226+0000 7fb592402700 1 -- start start 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.226+0000 7fb592402700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.226+0000 7fb592402700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c10e550 con 0x7fb58c101b40 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb58bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb58bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c10e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:41264/0 (socket says 172.21.15.186:41264) 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb58bfff700 1 -- 172.21.15.186:0/509765915 learned_addr learned my addr 172.21.15.186:0/509765915 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:52:58.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb58bfff700 1 -- 172.21.15.186:0/509765915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c005040 con 0x7fb58c101b40 2024-02-20T07:52:58.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb58bfff700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c10e010 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009b80 tx=0x7fb57c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:58.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57c00be50 con 0x7fb58c101b40 2024-02-20T07:52:58.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb57c009620 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.227+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58c10e750 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.228+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57c01c5b0 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.228+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58c10ebc0 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.228+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fb57c01c7e0 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.228+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58c0ffd50 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.229+0000 7fb588ff9700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb574036960 0x7fb574038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.229+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb57c049ce0 con 0x7fb58c101b40 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.229+0000 7fb58b7fe700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb574036960 0x7fb574038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:58.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.229+0000 7fb58b7fe700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb574036960 0x7fb574038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb580009990 tx=0x7fb580006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:58.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.234+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb57c023300 con 0x7fb58c101b40 2024-02-20T07:52:58.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.509+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb58c05f5c0 con 0x7fb58c101b40 2024-02-20T07:52:58.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.509+0000 7fb588ff9700 1 -- 172.21.15.186:0/509765915 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb57c011b60 con 0x7fb58c101b40 2024-02-20T07:52:58.511 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:52:58.511 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:52:58.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb574036960 msgr2=0x7fb574038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb574036960 0x7fb574038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb580009990 tx=0x7fb580006e40 comp rx=0 tx=0).stop 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 msgr2=0x7fb58c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c10e010 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009b80 tx=0x7fb57c009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 shutdown_connections 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb574036960 0x7fb574038e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 --2- 172.21.15.186:0/509765915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb58c101b40 0x7fb58c10e010 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 >> 172.21.15.186:0/509765915 conn(0x7fb58c0f7660 msgr2=0x7fb58c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 shutdown_connections 2024-02-20T07:52:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:52:58.511+0000 7fb592402700 1 -- 172.21.15.186:0/509765915 wait complete. 2024-02-20T07:52:58.514 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:52:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi186.shtgck", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi186.shtgck", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: Deploying daemon mgr.smithi186.shtgck on smithi186 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:52:58 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/509765915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:53:00.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:53:00.483 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:53:00.780 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:53:02.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:02.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:02.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:02.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:02.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:53:02.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:02.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:01 smithi055 ceph-mon[29254]: Deploying daemon mon.smithi186 on smithi186 2024-02-20T07:53:02.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 -- 172.21.15.186:0/3999249735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 msgr2=0x7f9490101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 --2- 172.21.15.186:0/3999249735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490101f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f948c0048c0 tx=0x7f948c009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 -- 172.21.15.186:0/3999249735 shutdown_connections 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 --2- 172.21.15.186:0/3999249735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490101f30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 -- 172.21.15.186:0/3999249735 >> 172.21.15.186:0/3999249735 conn(0x7f9490072850 msgr2=0x7f9490072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 -- 172.21.15.186:0/3999249735 shutdown_connections 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 -- 172.21.15.186:0/3999249735 wait complete. 2024-02-20T07:53:02.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 Processor -- start 2024-02-20T07:53:02.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.425+0000 7f9498353700 1 -- start start 2024-02-20T07:53:02.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f9498353700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:02.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f9498353700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9490113430 con 0x7f9490101b60 2024-02-20T07:53:02.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f94960ef700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:02.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f94960ef700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.186:41304/0 (socket says 172.21.15.186:41304) 2024-02-20T07:53:02.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f94960ef700 1 -- 172.21.15.186:0/470291231 learned_addr learned my addr 172.21.15.186:0/470291231 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:53:02.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f94960ef700 1 -- 172.21.15.186:0/470291231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c005040 con 0x7f9490101b60 2024-02-20T07:53:02.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.426+0000 7f94960ef700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490112ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f948c009300 tx=0x7f948c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:02.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.427+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948c004e60 con 0x7f9490101b60 2024-02-20T07:53:02.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.427+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9490113630 con 0x7f9490101b60 2024-02-20T07:53:02.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.427+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f948c013420 con 0x7f9490101b60 2024-02-20T07:53:02.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.427+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948c01c590 con 0x7f9490101b60 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.427+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9490113b20 con 0x7f9490101b60 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.428+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949005f5c0 con 0x7f9490101b60 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.428+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f948c009600 con 0x7f9490101b60 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.428+0000 7f9486ffd700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f947c036910 0x7f947c038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.428+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f948c049c70 con 0x7f9490101b60 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.428+0000 7f94958ee700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f947c036910 0x7f947c038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:02.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.429+0000 7f94958ee700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f947c036910 0x7f947c038dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9480009990 tx=0x7f9480006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:02.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.433+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f948c0119c0 con 0x7f9490101b60 2024-02-20T07:53:02.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.711+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9490105dc0 con 0x7f9490101b60 2024-02-20T07:53:02.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.712+0000 7f9486ffd700 1 -- 172.21.15.186:0/470291231 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f948c011370 con 0x7f9490101b60 2024-02-20T07:53:02.714 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:53:02.714 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:20.470988Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:53:02.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f947c036910 msgr2=0x7f947c038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:02.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f947c036910 0x7f947c038dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9480009990 tx=0x7f9480006e40 comp rx=0 tx=0).stop 2024-02-20T07:53:02.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 msgr2=0x7f9490112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:02.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490112ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f948c009300 tx=0x7f948c009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:02.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 shutdown_connections 2024-02-20T07:53:02.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f947c036910 0x7f947c038dc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:02.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 --2- 172.21.15.186:0/470291231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9490101b60 0x7f9490112ef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:02.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 >> 172.21.15.186:0/470291231 conn(0x7f9490072850 msgr2=0x7f94900ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:02.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 shutdown_connections 2024-02-20T07:53:02.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:02.714+0000 7f9498353700 1 -- 172.21.15.186:0/470291231 wait complete. 2024-02-20T07:53:02.718 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-02-20T07:53:03.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:02 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/470291231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:53:04.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:53:04.326 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:53:04.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:04 smithi055 ceph-mon[29254]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:04.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:04 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:04.862 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:53:06.259 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 podman[36731]: 2024-02-20 07:53:06.156658689 +0000 UTC m=+0.125952758 container create 6b7e709ad8c3f9c0bcdd56adbb3ad1067f6bda7688798d49ed76705350a70f46 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi186, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-20T07:53:06.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:06 smithi055 ceph-mon[29254]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:06.796 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 podman[36731]: 2024-02-20 07:53:06.536222801 +0000 UTC m=+0.505516864 container init 6b7e709ad8c3f9c0bcdd56adbb3ad1067f6bda7688798d49ed76705350a70f46 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi186, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD) 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 podman[36731]: 2024-02-20 07:53:06.540091024 +0000 UTC m=+0.509385087 container start 6b7e709ad8c3f9c0bcdd56adbb3ad1067f6bda7688798d49ed76705350a70f46 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi186, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: set uid:gid to 167:167 (ceph:ceph) 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable), process ceph-mon, pid 2 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: pidfile_write: ignore empty --pid-file 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: load: jerasure load: lrc 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: RocksDB version: 6.15.5 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Compile date Feb 19 2024 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: DB SUMMARY 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: DB Session ID: QVMQ1B0UHL6RMOIVVSE4 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: CURRENT file: CURRENT 2024-02-20T07:53:06.797 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: IDENTITY file: IDENTITY 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi186/store.db dir, Total Num: 0, files: 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi186/store.db: 000004.log size: 511 ; 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.error_if_exists: 0 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.create_if_missing: 0 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.paranoid_checks: 1 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.env: 0x555b85043a80 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.fs: Posix File System 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.info_log: 0x555b85de3cc0 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_file_opening_threads: 16 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.statistics: (nil) 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.use_fsync: 0 2024-02-20T07:53:06.798 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_log_file_size: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.keep_log_file_num: 1000 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.recycle_log_file_num: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.allow_fallocate: 1 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.allow_mmap_reads: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.allow_mmap_writes: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.use_direct_reads: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.create_missing_column_families: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.db_log_dir: 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi186/store.db 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T07:53:06.799 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.advise_random_on_open: 1 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.db_write_buffer_size: 0 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.write_buffer_manager: 0x555b86b58b40 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.rate_limiter: (nil) 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.wal_recovery_mode: 2 2024-02-20T07:53:06.800 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.enable_thread_tracking: 0 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.enable_pipelined_write: 0 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.unordered_write: 0 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.row_cache: None 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.wal_filter: None 2024-02-20T07:53:06.801 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T07:53:06.802 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.allow_ingest_behind: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.preserve_deletes: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.two_write_queues: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.manual_wal_flush: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.atomic_flush: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.log_readahead_size: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.best_efforts_recovery: 0 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-20T07:53:06.803 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.allow_data_in_errors: 0 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.db_host_id: __hostname__ 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_background_jobs: 2 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_background_compactions: -1 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_subcompactions: 1 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_total_wal_size: 0 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T07:53:06.804 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_open_files: -1 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bytes_per_sync: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_readahead_size: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_background_flushes: -1 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Compression algorithms supported: 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kZSTD supported: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kXpressCompression supported: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kLZ4HCCompression supported: 1 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kZlibCompression supported: 1 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kSnappyCompression supported: 1 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kLZ4Compression supported: 1 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: kBZip2Compression supported: 0 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T07:53:06.805 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi186/store.db/MANIFEST-000003 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.merge_operator: 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_filter: None 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_filter_factory: None 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.sst_partitioner_factory: None 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555b85e38f58) 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: index_type: 0 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: data_block_index_type: 0 2024-02-20T07:53:06.806 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: index_shortening: 1 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: hash_index_allow_collision: 1 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: checksum: 1 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: no_block_cache: 0 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_cache: 0x555b85e6b090 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_cache_options: 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: capacity : 536870912 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: num_shard_bits : 4 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: strict_capacity_limit : 0 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_cache_compressed: (nil) 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: persistent_cache: (nil) 2024-02-20T07:53:06.807 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_size: 4096 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_size_deviation: 10 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_restart_interval: 16 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: index_block_restart_interval: 1 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: metadata_block_size: 4096 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: partition_filters: 0 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: use_delta_encoding: 1 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: whole_key_filtering: 1 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: verify_compression: 0 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: format_version: 4 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: enable_index_compression: 1 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: block_align: 0 2024-02-20T07:53:06.808 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.write_buffer_size: 33554432 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_write_buffer_number: 2 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression: NoCompression 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression: Disabled 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.prefix_extractor: nullptr 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.num_levels: 7 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T07:53:06.809 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.level: 32767 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.strategy: 0 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compression_opts.enabled: false 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T07:53:06.810 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.target_file_size_base: 67108864 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T07:53:06.811 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.arena_block_size: 4194304 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.disable_auto_compactions: 0 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T07:53:06.812 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.table_properties_collectors: 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.inplace_update_support: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.bloom_locality: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.max_successive_merges: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.paranoid_file_checks: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.force_consistency_checks: 1 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.report_bg_io_stats: 0 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.ttl: 2592000 2024-02-20T07:53:06.813 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.enable_blob_files: false 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.min_blob_size: 0 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.blob_file_size: 268435456 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi186/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415586565911, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415586566212, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708415586, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "10680cb9-0645-4c99-af0c-36ce77f4ef9e", "db_session_id": "QVMQ1B0UHL6RMOIVVSE4"}} 2024-02-20T07:53:06.814 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415586566521, "job": 1, "event": "recovery_finished"} 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi186/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x555b85dff180 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: DB pointer 0x555b85ed4000 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** DB Stats ** 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T07:53:06.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** Compaction Stats [default] ** 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** Compaction Stats [default] ** 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:53:06.816 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** Compaction Stats [default] ** 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:53:06.817 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** Compaction Stats [default] ** 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T07:53:06.818 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186 does not exist in monmap, will attempt to join an existing cluster 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: using public_addr v2:172.21.15.186:0/0 -> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: starting mon.smithi186 rank -1 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi186 fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(???) e0 preinit fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).mds e1 new map 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).mds e1 print_map 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: e1 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:53:06.819 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: legacy client fscid: -1 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout: No filesystems configured 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:53:06.820 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:06.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-20T07:53:06.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 ceph-mon[36772]: mon.smithi186@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-20T07:53:06.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 bash[36731]: 6b7e709ad8c3f9c0bcdd56adbb3ad1067f6bda7688798d49ed76705350a70f46 2024-02-20T07:53:06.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:06 smithi186 systemd[1]: Started Ceph mon.smithi186 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:53:11.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:53:11.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:11.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: mon.smithi055 calling monitor election 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi186.shtgck/crt"}]: dispatch 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: mon.smithi186 calling monitor election 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: mon.smithi055 is new leader, mons smithi055,smithi186 in quorum (ranks 0,1) 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: monmap e2: 2 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],smithi186=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:53:11.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: fsmap 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: osdmap e5: 0 total, 0 up, 0 in 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: mgrmap e17: smithi055.stjper(active, since 28s) 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: Standby manager daemon smithi186.shtgck started 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: overall HEALTH_OK 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi186.shtgck/key"}]: dispatch 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:11.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:53:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: mon.smithi055 calling monitor election 2024-02-20T07:53:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi186.shtgck/crt"}]: dispatch 2024-02-20T07:53:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: mon.smithi186 calling monitor election 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: mon.smithi055 is new leader, mons smithi055,smithi186 in quorum (ranks 0,1) 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: monmap e2: 2 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],smithi186=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: fsmap 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: osdmap e5: 0 total, 0 up, 0 in 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: mgrmap e17: smithi055.stjper(active, since 28s) 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: Standby manager daemon smithi186.shtgck started 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-20T07:53:12.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: overall HEALTH_OK 2024-02-20T07:53:12.077 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi186.shtgck/key"}]: dispatch 2024-02-20T07:53:12.077 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.? 172.21.15.186:0/1705597029' entity='mgr.smithi186.shtgck' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-20T07:53:12.077 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:12.077 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:12.077 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:12.077 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:12.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 -- 172.21.15.186:0/285235300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 msgr2=0x7f650800a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:12.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 --2- 172.21.15.186:0/285235300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f650800a8c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f651c064f40 tx=0x7f6518002040 comp rx=0 tx=0).stop 2024-02-20T07:53:12.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 -- 172.21.15.186:0/285235300 shutdown_connections 2024-02-20T07:53:12.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 --2- 172.21.15.186:0/285235300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f650800a8c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:12.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 -- 172.21.15.186:0/285235300 >> 172.21.15.186:0/285235300 conn(0x7f651c072180 msgr2=0x7f651c072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 -- 172.21.15.186:0/285235300 shutdown_connections 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 -- 172.21.15.186:0/285235300 wait complete. 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.838+0000 7f65241ab700 1 Processor -- start 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f65241ab700 1 -- start start 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f65241ab700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f651c115310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f65241ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f65241ab700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c115e60 con 0x7f6508008460 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f65241ab700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c10f6f0 con 0x7f651c115850 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33746/0 (socket says 172.21.15.186:33746) 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 -- 172.21.15.186:0/2532790429 learned_addr learned my addr 172.21.15.186:0/2532790429 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521f47700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f651c115310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 -- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 msgr2=0x7f651c10f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 -- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 msgr2=0x7f651c10f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521746700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521f47700 1 -- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 msgr2=0x7f651c10f1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521f47700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.839+0000 7f6521f47700 1 -- 172.21.15.186:0/2532790429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6518005040 con 0x7f6508008460 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.840+0000 7f6521f47700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f651c115310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f651c064f40 tx=0x7f6518002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.840+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6518004290 con 0x7f6508008460 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.840+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6518018e30 con 0x7f6508008460 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.840+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c10f970 con 0x7f6508008460 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.840+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6518022c60 con 0x7f6508008460 2024-02-20T07:53:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.840+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c10fed0 con 0x7f6508008460 2024-02-20T07:53:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.841+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c05f5c0 con 0x7f6508008460 2024-02-20T07:53:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.842+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f6518009260 con 0x7f6508008460 2024-02-20T07:53:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.842+0000 7f65127fc700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6500069370 0x7f650006b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:12.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.842+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6518087d50 con 0x7f6508008460 2024-02-20T07:53:12.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.842+0000 7f6521746700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6500069370 0x7f650006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:12.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.843+0000 7f6521746700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6500069370 0x7f650006b820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f650c006fd0 tx=0x7f650c008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:12.847+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f651808b030 con 0x7f6508008460 2024-02-20T07:53:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:12 smithi055 ceph-mon[29254]: mgrmap e18: smithi055.stjper(active, since 28s), standbys: smithi186.shtgck 2024-02-20T07:53:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr metadata", "who": "smithi186.shtgck", "id": "smithi186.shtgck"}]: dispatch 2024-02-20T07:53:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:12 smithi186 ceph-mon[36772]: mgrmap e18: smithi055.stjper(active, since 28s), standbys: smithi186.shtgck 2024-02-20T07:53:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr metadata", "who": "smithi186.shtgck", "id": "smithi186.shtgck"}]: dispatch 2024-02-20T07:53:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:53:13.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.115+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f651c1108f0 con 0x7f6508008460 2024-02-20T07:53:13.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.116+0000 7f65127fc700 1 -- 172.21.15.186:0/2532790429 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f651802e090 con 0x7f6508008460 2024-02-20T07:53:13.117 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-02-20T07:53:13.117 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:53:06.591027Z","created":"2024-02-20T07:50:20.470988Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi186","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-20T07:53:13.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6500069370 msgr2=0x7f650006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6500069370 0x7f650006b820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f650c006fd0 tx=0x7f650c008040 comp rx=0 tx=0).stop 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 msgr2=0x7f651c115310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f651c115310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f651c064f40 tx=0x7f6518002040 comp rx=0 tx=0).stop 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 shutdown_connections 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6500069370 0x7f650006b820 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6508008460 0x7f651c115310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 --2- 172.21.15.186:0/2532790429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f651c115850 0x7f651c10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 >> 172.21.15.186:0/2532790429 conn(0x7f651c072180 msgr2=0x7f651c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 shutdown_connections 2024-02-20T07:53:13.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:53:13.118+0000 7f65241ab700 1 -- 172.21.15.186:0/2532790429 wait complete. 2024-02-20T07:53:13.121 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-02-20T07:53:13.710 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-20T07:53:13.710 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph config generate-minimal-conf 2024-02-20T07:53:13.980 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:53:14.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:13 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/2532790429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:53:14.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:13 smithi186 ceph-mon[36772]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:14.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:13 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:14.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:13 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:14.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:13 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/2532790429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:53:14.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:13 smithi055 ceph-mon[29254]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:14.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:13 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:14.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:13 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.373 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.373 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.373 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:15.373 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: Updating smithi055:/etc/ceph/ceph.conf 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: Updating smithi186:/etc/ceph/ceph.conf 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: Updating smithi186:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: Updating smithi055:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:53:15.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:15.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:15.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: Updating smithi055:/etc/ceph/ceph.conf 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: Updating smithi186:/etc/ceph/ceph.conf 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: Updating smithi186:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: Updating smithi055:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:53:15.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:16.322 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: Reconfiguring mon.smithi055 (unknown last config time)... 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: Reconfiguring daemon mon.smithi055 on smithi055 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi055.stjper", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:53:16.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:16 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 -- 172.21.15.55:0/3901781921 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 msgr2=0x7fd5cc0fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 --2- 172.21.15.55:0/3901781921 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fb9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc004940 tx=0x7fd5bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 -- 172.21.15.55:0/3901781921 shutdown_connections 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 --2- 172.21.15.55:0/3901781921 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fb9c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 --2- 172.21.15.55:0/3901781921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5cc103140 0x7fd5cc103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 -- 172.21.15.55:0/3901781921 >> 172.21.15.55:0/3901781921 conn(0x7fd5cc071560 msgr2=0x7fd5cc071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 -- 172.21.15.55:0/3901781921 shutdown_connections 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.536+0000 7fd5d3236700 1 -- 172.21.15.55:0/3901781921 wait complete. 2024-02-20T07:53:16.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d3236700 1 Processor -- start 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d3236700 1 -- start start 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d3236700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d3236700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5cc103140 0x7fd5cc0fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d3236700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0fd290 con 0x7fd5cc0fb570 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d3236700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0fd400 con 0x7fd5cc103140 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d0fd2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d0fd2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34870/0 (socket says 172.21.15.55:34870) 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5d0fd2700 1 -- 172.21.15.55:0/2759827267 learned_addr learned my addr 172.21.15.55:0/2759827267 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.537+0000 7fd5cbfff700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5cc103140 0x7fd5cc0fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5d0fd2700 1 -- 172.21.15.55:0/2759827267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5cc103140 msgr2=0x7fd5cc0fcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5d0fd2700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5cc103140 0x7fd5cc0fcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5d0fd2700 1 -- 172.21.15.55:0/2759827267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc005040 con 0x7fd5cc0fb570 2024-02-20T07:53:16.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5d0fd2700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fc750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400cac0 tx=0x7fd5c4007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c400aa50 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5c400abb0 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc119dd0 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c40115e0 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.538+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc11a2b0 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.539+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc1054d0 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.540+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fd5c400ad20 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.540+0000 7fd5c97fa700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd5b8069210 0x7fd5b806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.540+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd5c4086750 con 0x7fd5cc0fb570 2024-02-20T07:53:16.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.540+0000 7fd5cbfff700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd5b8069210 0x7fd5b806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:16.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.541+0000 7fd5cbfff700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd5b8069210 0x7fd5b806b6c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00b3b0 tx=0x7fd5bc009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:16.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.548+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd5c4053b60 con 0x7fd5cc0fb570 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: Reconfiguring mon.smithi055 (unknown last config time)... 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: Reconfiguring daemon mon.smithi055 on smithi055 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi055.stjper", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:53:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:16 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:16.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.744+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd5cc073080 con 0x7fd5cc0fb570 2024-02-20T07:53:16.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.744+0000 7fd5c97fa700 1 -- 172.21.15.55:0/2759827267 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fd5c4085f00 con 0x7fd5cc0fb570 2024-02-20T07:53:16.746 INFO:teuthology.orchestra.run.smithi055.stdout:# minimal ceph.conf for 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:53:16.746 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2024-02-20T07:53:16.746 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = 9a1162a4-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:53:16.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon_host = [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd5b8069210 msgr2=0x7fd5b806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd5b8069210 0x7fd5b806b6c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00b3b0 tx=0x7fd5bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 msgr2=0x7fd5cc0fc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fc750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400cac0 tx=0x7fd5c4007ee0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 shutdown_connections 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd5b8069210 0x7fd5b806b6c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5cc0fb570 0x7fd5cc0fc750 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 --2- 172.21.15.55:0/2759827267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5cc103140 0x7fd5cc0fcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 >> 172.21.15.55:0/2759827267 conn(0x7fd5cc071560 msgr2=0x7fd5cc0f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.746+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 shutdown_connections 2024-02-20T07:53:16.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:16.747+0000 7fd5d3236700 1 -- 172.21.15.55:0/2759827267 wait complete. 2024-02-20T07:53:17.622 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:17 smithi055 ceph-mon[29254]: Reconfiguring mgr.smithi055.stjper (unknown last config time)... 2024-02-20T07:53:17.622 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:17 smithi055 ceph-mon[29254]: Reconfiguring daemon mgr.smithi055.stjper on smithi055 2024-02-20T07:53:17.622 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:17 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2759827267' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:17.878 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-20T07:53:17.878 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:53:17.879 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.conf 2024-02-20T07:53:17.907 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:53:17.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:53:17.974 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:53:17.974 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-02-20T07:53:18.003 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:17 smithi186 ceph-mon[36772]: Reconfiguring mgr.smithi055.stjper (unknown last config time)... 2024-02-20T07:53:18.004 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:17 smithi186 ceph-mon[36772]: Reconfiguring daemon mgr.smithi055.stjper on smithi055 2024-02-20T07:53:18.004 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:17 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2759827267' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:18.009 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:53:18.009 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:53:18.080 INFO:tasks.cephadm:Deploying OSDs... 2024-02-20T07:53:18.080 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:53:18.080 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-02-20T07:53:18.095 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-20T07:53:18.095 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-02-20T07:53:18.149 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-20T07:53:18.149 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:18.149 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-20T07:53:18.149 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:18.149 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:18.150 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-02-20 07:52:34.897685284 +0000 2024-02-20T07:53:18.150 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-02-20 07:46:34.786246315 +0000 2024-02-20T07:53:18.150 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-02-20 07:46:34.786246315 +0000 2024-02-20T07:53:18.150 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-02-20 07:46:34.786246315 +0000 2024-02-20T07:53:18.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-20T07:53:18.214 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-02-20T07:53:18.214 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-02-20T07:53:18.214 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000265111 s, 1.9 MB/s 2024-02-20T07:53:18.215 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-20T07:53:18.271 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-02-20 07:52:34.898685265 +0000 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-02-20 07:46:35.135239420 +0000 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-02-20 07:46:35.135239420 +0000 2024-02-20T07:53:18.327 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-02-20 07:46:35.135239420 +0000 2024-02-20T07:53:18.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-20T07:53:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-02-20T07:53:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-02-20T07:53:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000265369 s, 1.9 MB/s 2024-02-20T07:53:18.395 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-20T07:53:18.450 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-02-20T07:53:18.505 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-20T07:53:18.505 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:18.505 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-20T07:53:18.505 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:18.505 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:18.506 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-02-20 07:52:34.898685265 +0000 2024-02-20T07:53:18.506 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-02-20 07:46:35.483232546 +0000 2024-02-20T07:53:18.506 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-02-20 07:46:35.483232546 +0000 2024-02-20T07:53:18.506 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-02-20 07:46:35.483232546 +0000 2024-02-20T07:53:18.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-20T07:53:18.571 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-02-20T07:53:18.571 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-02-20T07:53:18.571 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000267968 s, 1.9 MB/s 2024-02-20T07:53:18.572 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-20T07:53:18.628 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-02-20T07:53:18.684 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-20T07:53:18.684 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:18.684 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-20T07:53:18.684 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:18.685 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:18.685 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-02-20 07:52:34.898685265 +0000 2024-02-20T07:53:18.685 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-02-20 07:46:35.830225692 +0000 2024-02-20T07:53:18.685 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-02-20 07:46:35.830225692 +0000 2024-02-20T07:53:18.685 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-02-20 07:46:35.830225692 +0000 2024-02-20T07:53:18.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-20T07:53:18.750 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-02-20T07:53:18.750 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-02-20T07:53:18.750 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000268282 s, 1.9 MB/s 2024-02-20T07:53:18.751 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-20T07:53:18.806 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:53:18.807 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-02-20T07:53:18.826 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-20T07:53:18.826 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-02-20T07:53:18.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:18 smithi055 ceph-mon[29254]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:18.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:18 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:18.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:18 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:18.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:18 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:53:18.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:18 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-02-20 07:53:05.103371285 +0000 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-02-20 07:46:32.740624701 +0000 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-02-20 07:46:32.740624701 +0000 2024-02-20T07:53:18.884 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-02-20 07:46:32.740624701 +0000 2024-02-20T07:53:18.885 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-20T07:53:18.951 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:18 smithi186 ceph-mon[36772]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:18.951 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:18 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:18.951 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:18 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:18.951 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:18 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:53:18.952 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:18 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:18.954 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-02-20T07:53:18.954 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-02-20T07:53:18.954 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000209957 s, 2.4 MB/s 2024-02-20T07:53:18.955 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-20T07:53:19.014 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-02-20T07:53:19.072 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-20T07:53:19.072 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-02-20 07:53:05.103371285 +0000 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-02-20 07:46:33.088618404 +0000 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-02-20 07:46:33.088618404 +0000 2024-02-20T07:53:19.073 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-02-20 07:46:33.088618404 +0000 2024-02-20T07:53:19.073 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-20T07:53:19.142 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-02-20T07:53:19.142 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-02-20T07:53:19.142 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00022592 s, 2.3 MB/s 2024-02-20T07:53:19.143 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-20T07:53:19.201 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-02-20T07:53:19.258 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-20T07:53:19.258 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:19.258 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-20T07:53:19.259 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:19.259 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:19.259 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-02-20 07:53:05.103371285 +0000 2024-02-20T07:53:19.259 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-02-20 07:46:33.434612144 +0000 2024-02-20T07:53:19.259 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-02-20 07:46:33.434612144 +0000 2024-02-20T07:53:19.259 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-02-20 07:46:33.434612144 +0000 2024-02-20T07:53:19.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-20T07:53:19.327 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-02-20T07:53:19.328 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-02-20T07:53:19.328 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000208122 s, 2.5 MB/s 2024-02-20T07:53:19.329 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-20T07:53:19.386 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-02-20T07:53:19.442 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-20T07:53:19.442 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:53:19.442 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-20T07:53:19.443 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:53:19.443 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:53:19.443 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-02-20 07:53:05.103371285 +0000 2024-02-20T07:53:19.443 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-02-20 07:46:33.765606154 +0000 2024-02-20T07:53:19.443 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-02-20 07:46:33.765606154 +0000 2024-02-20T07:53:19.443 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-02-20 07:46:33.765606154 +0000 2024-02-20T07:53:19.443 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-20T07:53:19.510 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-02-20T07:53:19.510 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-02-20T07:53:19.511 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000266146 s, 1.9 MB/s 2024-02-20T07:53:19.512 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-20T07:53:19.569 INFO:tasks.cephadm:Deploying osd.0 on smithi055 with /dev/vg_nvme/lv_4... 2024-02-20T07:53:19.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-20T07:53:19.837 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:53:19.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:19 smithi055 ceph-mon[29254]: Reconfiguring crash.smithi055 (monmap changed)... 2024-02-20T07:53:19.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:19 smithi055 ceph-mon[29254]: Reconfiguring daemon crash.smithi055 on smithi055 2024-02-20T07:53:20.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:19 smithi186 ceph-mon[36772]: Reconfiguring crash.smithi055 (monmap changed)... 2024-02-20T07:53:20.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:19 smithi186 ceph-mon[36772]: Reconfiguring daemon crash.smithi055 on smithi055 2024-02-20T07:53:20.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:20 smithi055 ceph-mon[29254]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:20.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:20 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:20.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:20 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:21.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:20 smithi186 ceph-mon[36772]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:21.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:20 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:21.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:20 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:21.651 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:21 smithi055 ceph-mon[29254]: Reconfiguring alertmanager.smithi055 (dependencies changed)... 2024-02-20T07:53:21.651 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:21 smithi055 ceph-mon[29254]: Reconfiguring daemon alertmanager.smithi055 on smithi055 2024-02-20T07:53:22.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:21 smithi186 ceph-mon[36772]: Reconfiguring alertmanager.smithi055 (dependencies changed)... 2024-02-20T07:53:22.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:21 smithi186 ceph-mon[36772]: Reconfiguring daemon alertmanager.smithi055 on smithi055 2024-02-20T07:53:22.917 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:22 smithi055 ceph-mon[29254]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:22.966 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-20T07:53:22.966 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-20T07:53:22.966 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-02-20T07:53:22.966 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-02-20T07:53:22.966 INFO:teuthology.orchestra.run.smithi055.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015585 s, 673 MB/s 2024-02-20T07:53:22.967 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-02-20T07:53:23.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:22 smithi186 ceph-mon[36772]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:24.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:23 smithi055 ceph-mon[29254]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:24.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:23 smithi186 ceph-mon[36772]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:24.578 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_4 2024-02-20T07:53:25.494 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:53:26.555 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:26 smithi055 ceph-mon[29254]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:26.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:26 smithi186 ceph-mon[36772]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 -- 172.21.15.55:0/2082572740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 msgr2=0x7ff7b80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 --2- 172.21.15.55:0/2082572740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b80fd420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac004940 tx=0x7ff7ac009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 -- 172.21.15.55:0/2082572740 shutdown_connections 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 --2- 172.21.15.55:0/2082572740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b80fd420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 --2- 172.21.15.55:0/2082572740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7b80fbe40 0x7ff7b80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 -- 172.21.15.55:0/2082572740 >> 172.21.15.55:0/2082572740 conn(0x7ff7b80f7520 msgr2=0x7ff7b80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 -- 172.21.15.55:0/2082572740 shutdown_connections 2024-02-20T07:53:26.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.859+0000 7ff7be368700 1 -- 172.21.15.55:0/2082572740 wait complete. 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7be368700 1 Processor -- start 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7be368700 1 -- start start 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7be368700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7b80fbe40 0x7ff7b810d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7be368700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7be368700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b810e4e0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7be368700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b810ed20 con 0x7ff7b80fbe40 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7b77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7b77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b810de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48732/0 (socket says 172.21.15.55:48732) 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.860+0000 7ff7b77fe700 1 -- 172.21.15.55:0/3487124015 learned_addr learned my addr 172.21.15.55:0/3487124015 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b7fff700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7b80fbe40 0x7ff7b810d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b77fe700 1 -- 172.21.15.55:0/3487124015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7b80fbe40 msgr2=0x7ff7b810d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:26.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b77fe700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7b80fbe40 0x7ff7b810d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b77fe700 1 -- 172.21.15.55:0/3487124015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac005040 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b77fe700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b810de50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0093b0 tx=0x7ff7ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7ac00e040 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7ac00bcc0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7ac01c630 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b810efa0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.861+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b810f4e0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.862+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b804bed0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.862+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7ff7ac0139f0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.863+0000 7ff7b4ff9700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff7a0069210 0x7ff7a006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:26.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.863+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff7ac0873b0 con 0x7ff7b80fcfb0 2024-02-20T07:53:26.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.863+0000 7ff7b7fff700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff7a0069210 0x7ff7a006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:26.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.863+0000 7ff7b7fff700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff7a0069210 0x7ff7a006b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8006fd0 tx=0x7ff7a8008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:26.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:26.867+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff7ac0582f0 con 0x7ff7b80fcfb0 2024-02-20T07:53:27.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:27.073+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7b810f9c0 con 0x7ff7a0069210 2024-02-20T07:53:27.464 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:27 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:53:27.464 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:27 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:53:27.464 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:27 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:27.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:27 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:53:27.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:27 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:53:27.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:27 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:28.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:28 smithi055 ceph-mon[29254]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:53:28.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:28 smithi055 ceph-mon[29254]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:28.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:28.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:28.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:28 smithi055 ceph-mon[29254]: Reconfiguring grafana.smithi055 (dependencies changed)... 2024-02-20T07:53:28.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:28 smithi186 ceph-mon[36772]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:53:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:28 smithi186 ceph-mon[36772]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:29.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:28 smithi186 ceph-mon[36772]: Reconfiguring grafana.smithi055 (dependencies changed)... 2024-02-20T07:53:29.076 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:29.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:29 smithi055 ceph-mon[29254]: Reconfiguring daemon grafana.smithi055 on smithi055 2024-02-20T07:53:30.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:29 smithi186 ceph-mon[36772]: Reconfiguring daemon grafana.smithi055 on smithi055 2024-02-20T07:53:30.694 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:30 smithi055 ceph-mon[29254]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:31.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:30 smithi186 ceph-mon[36772]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:32.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:32 smithi055 ceph-mon[29254]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:32.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:32 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:32.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:32 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:33.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:32 smithi186 ceph-mon[36772]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:33.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:32 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:33.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:32 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:33 smithi055 ceph-mon[29254]: Reconfiguring prometheus.smithi055 (dependencies changed)... 2024-02-20T07:53:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:33 smithi055 ceph-mon[29254]: Reconfiguring daemon prometheus.smithi055 on smithi055 2024-02-20T07:53:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:33 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/413320203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d58e18-643f-4fc7-bb34-76cb0ba30b3d"}]: dispatch 2024-02-20T07:53:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:33 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/413320203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16d58e18-643f-4fc7-bb34-76cb0ba30b3d"}]': finished 2024-02-20T07:53:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:33 smithi055 ceph-mon[29254]: osdmap e6: 1 total, 0 up, 1 in 2024-02-20T07:53:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:33 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:34.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:33 smithi186 ceph-mon[36772]: Reconfiguring prometheus.smithi055 (dependencies changed)... 2024-02-20T07:53:34.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:33 smithi186 ceph-mon[36772]: Reconfiguring daemon prometheus.smithi055 on smithi055 2024-02-20T07:53:34.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:33 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/413320203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d58e18-643f-4fc7-bb34-76cb0ba30b3d"}]: dispatch 2024-02-20T07:53:34.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:33 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/413320203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16d58e18-643f-4fc7-bb34-76cb0ba30b3d"}]': finished 2024-02-20T07:53:34.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:33 smithi186 ceph-mon[36772]: osdmap e6: 1 total, 0 up, 1 in 2024-02-20T07:53:34.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:33 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:35.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:34 smithi186 ceph-mon[36772]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:35.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:34 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2048743841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:53:35.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:34 smithi055 ceph-mon[29254]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:35.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:34 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2048743841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:53:36.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:35 smithi186 ceph-mon[36772]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:36.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:35 smithi055 ceph-mon[29254]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:38.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:38 smithi186 ceph-mon[36772]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:38.709 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:38 smithi055 ceph-mon[29254]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:40.550 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:40 smithi055 ceph-mon[29254]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:40.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:40 smithi186 ceph-mon[36772]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:42 smithi186 ceph-mon[36772]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:42.655 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:42 smithi055 ceph-mon[29254]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:44 smithi055 ceph-mon[29254]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:44 smithi186 ceph-mon[36772]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:45.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:45.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:45.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:45 smithi055 ceph-mon[29254]: Reconfiguring crash.smithi186 (monmap changed)... 2024-02-20T07:53:45.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi186", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:53:45.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:45.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:45 smithi055 ceph-mon[29254]: Reconfiguring daemon crash.smithi186 on smithi186 2024-02-20T07:53:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:45 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:45 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:45 smithi186 ceph-mon[36772]: Reconfiguring crash.smithi186 (monmap changed)... 2024-02-20T07:53:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:45 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi186", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:53:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:45 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:45 smithi186 ceph-mon[36772]: Reconfiguring daemon crash.smithi186 on smithi186 2024-02-20T07:53:46.483 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:46.484 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:46.484 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:46.484 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: Reconfiguring mgr.smithi186.shtgck (monmap changed)... 2024-02-20T07:53:46.484 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi186.shtgck", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:53:46.484 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:53:46.485 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:46.485 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: Reconfiguring daemon mgr.smithi186.shtgck on smithi186 2024-02-20T07:53:46.485 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-20T07:53:46.485 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:46.485 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:46 smithi186 ceph-mon[36772]: Deploying daemon osd.0 on smithi055 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: Reconfiguring mgr.smithi186.shtgck (monmap changed)... 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi186.shtgck", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: Reconfiguring daemon mgr.smithi186.shtgck on smithi186 2024-02-20T07:53:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-20T07:53:46.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:46.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:46 smithi055 ceph-mon[29254]: Deploying daemon osd.0 on smithi055 2024-02-20T07:53:47.928 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:47.928 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:47.928 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:53:47.929 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:53:47.929 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:48.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:53:48.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:53:48.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:48.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: Reconfiguring mon.smithi186 (monmap changed)... 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: Reconfiguring daemon mon.smithi186 on smithi186 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:53:48.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.807 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:48 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: Reconfiguring mon.smithi186 (monmap changed)... 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: Reconfiguring daemon mon.smithi186 on smithi186 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:48.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:48 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:50.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:50 smithi055 ceph-mon[29254]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:50.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:50.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:50 smithi186 ceph-mon[36772]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:52.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:52 smithi055 ceph-mon[29254]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:52.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:52 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:52.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:52 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:53.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:52 smithi186 ceph-mon[36772]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:53.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:52 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:53.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:52 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:54.088 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:54 smithi055 ceph-mon[29254]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:54.088 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:54.088 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:54.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:54 smithi186 ceph-mon[36772]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:54.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:54 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:54.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:54 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:55.191 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:55 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:55.191 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:55 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:55.192 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:55 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:55.192 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:55 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:55.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:55 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:55.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:55 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:55.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:55 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:55.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:55 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:56.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:56 smithi186 ceph-mon[36772]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:56.370 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:56 smithi055 ceph-mon[29254]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:56.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.443+0000 7ff7b4ff9700 1 -- 172.21.15.55:0/3487124015 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7b810f9c0 con 0x7ff7a0069210 2024-02-20T07:53:56.445 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 0 on host 'smithi055' 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff7a0069210 msgr2=0x7ff7a006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff7a0069210 0x7ff7a006b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8006fd0 tx=0x7ff7a8008040 comp rx=0 tx=0).stop 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 msgr2=0x7ff7b810de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b810de50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0093b0 tx=0x7ff7ac009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 shutdown_connections 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7b80fbe40 0x7ff7b810d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff7a0069210 0x7ff7a006b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:56.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 --2- 172.21.15.55:0/3487124015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7b80fcfb0 0x7ff7b810de50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:56.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.445+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 >> 172.21.15.55:0/3487124015 conn(0x7ff7b80f7520 msgr2=0x7ff7b8065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:56.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.446+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 shutdown_connections 2024-02-20T07:53:56.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:53:56.446+0000 7ff7be368700 1 -- 172.21.15.55:0/3487124015 wait complete. 2024-02-20T07:53:57.217 DEBUG:teuthology.orchestra.run.smithi055:osd.0> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.0.service 2024-02-20T07:53:57.220 INFO:tasks.cephadm:Deploying osd.1 on smithi055 with /dev/vg_nvme/lv_3... 2024-02-20T07:53:57.220 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-20T07:53:57.287 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:57 smithi055 ceph-mon[29254]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-20T07:53:57.287 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:57 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.287 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:57 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.288 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:57 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.288 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:57 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:57 smithi186 ceph-mon[36772]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-20T07:53:57.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:57 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:57 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:57 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:57.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:57 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:53:58.020 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:53:58.276 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-mon[29254]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-20T07:53:58.276 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-mon[29254]: osdmap e7: 1 total, 0 up, 1 in 2024-02-20T07:53:58.276 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:58.276 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-mon[29254]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-02-20T07:53:58.276 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-mon[29254]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:58.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:53:58.026+0000 7fcd371fd700 -1 osd.0 0 waiting for initial osdmap 2024-02-20T07:53:58.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:53:58.031+0000 7fcd327f9700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:53:58.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:58 smithi186 ceph-mon[36772]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-20T07:53:58.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:58 smithi186 ceph-mon[36772]: osdmap e7: 1 total, 0 up, 1 in 2024-02-20T07:53:58.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:58.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:58 smithi186 ceph-mon[36772]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-02-20T07:53:58.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:58 smithi186 ceph-mon[36772]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:53:59.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:59 smithi186 ceph-mon[36772]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-02-20T07:53:59.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:59 smithi186 ceph-mon[36772]: osdmap e8: 1 total, 0 up, 1 in 2024-02-20T07:53:59.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:59 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:59.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:59 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:59.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:53:59 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:59.465 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:59 smithi055 ceph-mon[29254]: from='osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-02-20T07:53:59.465 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:59 smithi055 ceph-mon[29254]: osdmap e8: 1 total, 0 up, 1 in 2024-02-20T07:53:59.465 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:59 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:59.465 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:59 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:53:59.465 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:53:59 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:00.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:00 smithi186 ceph-mon[36772]: purged_snaps scrub starts 2024-02-20T07:54:00.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:00 smithi186 ceph-mon[36772]: purged_snaps scrub ok 2024-02-20T07:54:00.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:00 smithi186 ceph-mon[36772]: osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] boot 2024-02-20T07:54:00.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:00 smithi186 ceph-mon[36772]: osdmap e9: 1 total, 1 up, 1 in 2024-02-20T07:54:00.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:54:00.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:00 smithi186 ceph-mon[36772]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:54:00.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:00 smithi055 ceph-mon[29254]: purged_snaps scrub starts 2024-02-20T07:54:00.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:00 smithi055 ceph-mon[29254]: purged_snaps scrub ok 2024-02-20T07:54:00.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:00 smithi055 ceph-mon[29254]: osd.0 [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] boot 2024-02-20T07:54:00.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:00 smithi055 ceph-mon[29254]: osdmap e9: 1 total, 1 up, 1 in 2024-02-20T07:54:00.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:54:00.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:00 smithi055 ceph-mon[29254]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:54:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:01 smithi055 ceph-mon[29254]: osdmap e10: 1 total, 1 up, 1 in 2024-02-20T07:54:01.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:01 smithi186 ceph-mon[36772]: osdmap e10: 1 total, 1 up, 1 in 2024-02-20T07:54:01.387 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-20T07:54:01.387 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-20T07:54:01.387 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-02-20T07:54:01.387 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-02-20T07:54:01.387 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0160059 s, 655 MB/s 2024-02-20T07:54:01.387 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-02-20T07:54:02.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:02 smithi186 ceph-mon[36772]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:02.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:02.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:02.339 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:02 smithi055 ceph-mon[29254]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:02.339 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:02.339 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:02.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_3 2024-02-20T07:54:03.669 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:54:04.266 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:04 smithi055 ceph-mon[29254]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:04 smithi186 ceph-mon[36772]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 -- 172.21.15.55:0/1376020806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 msgr2=0x7fe5540fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1376020806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe5540fb9f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe544004940 tx=0x7fe544009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 -- 172.21.15.55:0/1376020806 shutdown_connections 2024-02-20T07:54:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1376020806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5540fc7f0 0x7fe5540fcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1376020806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe5540fb9f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 -- 172.21.15.55:0/1376020806 >> 172.21.15.55:0/1376020806 conn(0x7fe5540f6ca0 msgr2=0x7fe5540f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 -- 172.21.15.55:0/1376020806 shutdown_connections 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 -- 172.21.15.55:0/1376020806 wait complete. 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 Processor -- start 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 -- start start 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.172+0000 7fe5598fa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe55410da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe5598fa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5540fc7f0 0x7fe55410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe5598fa700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55410e600 con 0x7fe5540fb5e0 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe5598fa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55410ee40 con 0x7fe5540fc7f0 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe55410da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe55410da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43430/0 (socket says 172.21.15.55:43430) 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 -- 172.21.15.55:0/1513795131 learned_addr learned my addr 172.21.15.55:0/1513795131 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe5527fc700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5540fc7f0 0x7fe55410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 -- 172.21.15.55:0/1513795131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5540fc7f0 msgr2=0x7fe55410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5540fc7f0 0x7fe55410df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 -- 172.21.15.55:0/1513795131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe544005040 con 0x7fe5540fb5e0 2024-02-20T07:54:06.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe552ffd700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe55410da30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe544009390 tx=0x7fe544009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54400e040 con 0x7fe5540fb5e0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe544004290 con 0x7fe5540fb5e0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.173+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54401c3f0 con 0x7fe5540fb5e0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.174+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55410f0c0 con 0x7fe5540fb5e0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.174+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55410f5b0 con 0x7fe5540fb5e0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.175+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55404bed0 con 0x7fe5540fb5e0 2024-02-20T07:54:06.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.175+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fe544013420 con 0x7fe5540fb5e0 2024-02-20T07:54:06.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.176+0000 7fe53b7fe700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe53c069210 0x7fe53c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:06.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.176+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fe5440875a0 con 0x7fe5540fb5e0 2024-02-20T07:54:06.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.176+0000 7fe5527fc700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe53c069210 0x7fe53c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:06.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.177+0000 7fe5527fc700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe53c069210 0x7fe53c06b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe548009bf0 tx=0x7fe5480091a0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:06.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.184+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe5440581e0 con 0x7fe5540fb5e0 2024-02-20T07:54:06.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:06.386+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe55410fa20 con 0x7fe53c069210 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: Detected new or changed devices on smithi055 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: Adjusting osd_memory_target on smithi055 to 12869M 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:06.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:06 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: Detected new or changed devices on smithi055 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: Adjusting osd_memory_target on smithi055 to 12869M 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:06.453 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:06.454 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:06 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:07.346 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:07 smithi055 ceph-mon[29254]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:07.346 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:07 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:54:07.346 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:07 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:54:07.346 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:07 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:07.346 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:07 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:07.346 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:07 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:07.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:07 smithi186 ceph-mon[36772]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:07.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:07 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:54:07.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:07 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:54:07.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:07 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:07.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:07 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:07.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:07 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:08.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:08 smithi055 ceph-mon[29254]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:08.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:08 smithi186 ceph-mon[36772]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:10.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:10 smithi055 ceph-mon[29254]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:10.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:10 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2028917082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "089fab11-e47a-425b-83d8-10bc9dce41e7"}]: dispatch 2024-02-20T07:54:10.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:10 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2028917082' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "089fab11-e47a-425b-83d8-10bc9dce41e7"}]': finished 2024-02-20T07:54:10.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:10 smithi055 ceph-mon[29254]: osdmap e11: 2 total, 1 up, 2 in 2024-02-20T07:54:10.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:10 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:10.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:10 smithi186 ceph-mon[36772]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:10.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:10 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2028917082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "089fab11-e47a-425b-83d8-10bc9dce41e7"}]: dispatch 2024-02-20T07:54:10.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:10 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2028917082' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "089fab11-e47a-425b-83d8-10bc9dce41e7"}]': finished 2024-02-20T07:54:10.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:10 smithi186 ceph-mon[36772]: osdmap e11: 2 total, 1 up, 2 in 2024-02-20T07:54:10.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:10 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:11.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:11 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1568096843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:54:12.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:11 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1568096843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:54:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:12 smithi055 ceph-mon[29254]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:12.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:12.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:12.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:12.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:12.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:12 smithi186 ceph-mon[36772]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:13.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:14.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:14 smithi186 ceph-mon[36772]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:14.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:14 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:14.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:14 smithi055 ceph-mon[29254]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:14.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:14 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:16.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:16 smithi186 ceph-mon[36772]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:16.667 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:16 smithi055 ceph-mon[29254]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:18.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:18 smithi186 ceph-mon[36772]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:18.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:18 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-20T07:54:18.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:18 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:18.593 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:18 smithi055 ceph-mon[29254]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:18.593 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:18 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-20T07:54:18.593 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:18 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:19.642 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:19 smithi055 ceph-mon[29254]: Deploying daemon osd.1 on smithi055 2024-02-20T07:54:19.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:19 smithi186 ceph-mon[36772]: Deploying daemon osd.1 on smithi055 2024-02-20T07:54:20.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:20 smithi055 ceph-mon[29254]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:20.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:20 smithi186 ceph-mon[36772]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:22.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:22 smithi186 ceph-mon[36772]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:22.654 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:22 smithi055 ceph-mon[29254]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:24 smithi055 ceph-mon[29254]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:24.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:24.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:24.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:24.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:24 smithi186 ceph-mon[36772]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:26.424 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:26 smithi055 ceph-mon[29254]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:26.493 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 1 on host 'smithi055' 2024-02-20T07:54:26.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.491+0000 7fe53b7fe700 1 -- 172.21.15.55:0/1513795131 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe55410fa20 con 0x7fe53c069210 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe53c069210 msgr2=0x7fe53c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe53c069210 0x7fe53c06b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe548009bf0 tx=0x7fe5480091a0 comp rx=0 tx=0).stop 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 msgr2=0x7fe55410da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe55410da30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe544009390 tx=0x7fe544009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 shutdown_connections 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe53c069210 0x7fe53c06b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:26.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5540fb5e0 0x7fe55410da30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:26.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 --2- 172.21.15.55:0/1513795131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5540fc7f0 0x7fe55410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:26.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 >> 172.21.15.55:0/1513795131 conn(0x7fe5540f6ca0 msgr2=0x7fe554101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:26.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 shutdown_connections 2024-02-20T07:54:26.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:26.494+0000 7fe5598fa700 1 -- 172.21.15.55:0/1513795131 wait complete. 2024-02-20T07:54:26.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:26 smithi186 ceph-mon[36772]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:27.110 DEBUG:teuthology.orchestra.run.smithi055:osd.1> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.1.service 2024-02-20T07:54:27.113 INFO:tasks.cephadm:Deploying osd.2 on smithi055 with /dev/vg_nvme/lv_2... 2024-02-20T07:54:27.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-20T07:54:27.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:27 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:27.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:27 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:27.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:27 smithi055 ceph-mon[29254]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-20T07:54:27.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:27 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:27.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:27 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:27.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:27 smithi186 ceph-mon[36772]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-20T07:54:27.853 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:54:28.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-mon[29254]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:28.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-mon[29254]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-20T07:54:28.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-mon[29254]: osdmap e12: 2 total, 1 up, 2 in 2024-02-20T07:54:28.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:28.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-mon[29254]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-02-20T07:54:28.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:28.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:54:28.469+0000 7f0db5ba5700 -1 osd.1 0 waiting for initial osdmap 2024-02-20T07:54:28.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:54:28.476+0000 7f0db09a0700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:54:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:28 smithi186 ceph-mon[36772]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:28 smithi186 ceph-mon[36772]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-20T07:54:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:28 smithi186 ceph-mon[36772]: osdmap e12: 2 total, 1 up, 2 in 2024-02-20T07:54:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:28 smithi186 ceph-mon[36772]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-02-20T07:54:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:29.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:29 smithi055 ceph-mon[29254]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-02-20T07:54:29.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:29 smithi055 ceph-mon[29254]: osdmap e13: 2 total, 1 up, 2 in 2024-02-20T07:54:29.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:29 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:29.584 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:29 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:29.827 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:29 smithi186 ceph-mon[36772]: from='osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-02-20T07:54:29.827 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:29 smithi186 ceph-mon[36772]: osdmap e13: 2 total, 1 up, 2 in 2024-02-20T07:54:29.828 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:29 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:29.828 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:29 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-20T07:54:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-20T07:54:30.420 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-02-20T07:54:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-02-20T07:54:30.420 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128134 s, 818 MB/s 2024-02-20T07:54:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: purged_snaps scrub starts 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: purged_snaps scrub ok 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] boot 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: osdmap e14: 2 total, 2 up, 2 in 2024-02-20T07:54:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:30 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: purged_snaps scrub starts 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: purged_snaps scrub ok 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: osd.1 [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] boot 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: osdmap e14: 2 total, 2 up, 2 in 2024-02-20T07:54:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:30 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:54:31.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_2 2024-02-20T07:54:31.573 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:31 smithi055 ceph-mon[29254]: osdmap e15: 2 total, 2 up, 2 in 2024-02-20T07:54:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:31 smithi186 ceph-mon[36772]: osdmap e15: 2 total, 2 up, 2 in 2024-02-20T07:54:32.012 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:54:32.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:32 smithi055 ceph-mon[29254]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:32.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:32 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:32.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:32 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:32.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:32 smithi186 ceph-mon[36772]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:32.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:32 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:32.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:32 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 -- 172.21.15.55:0/2474565817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf01000c0 msgr2=0x7f9bf0102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/2474565817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf01000c0 0x7f9bf0102520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9be4004940 tx=0x7f9be4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 -- 172.21.15.55:0/2474565817 shutdown_connections 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/2474565817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf01000c0 0x7f9bf0102520 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/2474565817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00663e0 0x7f9bf00ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 -- 172.21.15.55:0/2474565817 >> 172.21.15.55:0/2474565817 conn(0x7f9bf00f74e0 msgr2=0x7f9bf00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 -- 172.21.15.55:0/2474565817 shutdown_connections 2024-02-20T07:54:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 -- 172.21.15.55:0/2474565817 wait complete. 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.640+0000 7f9bf4a95700 1 Processor -- start 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bf4a95700 1 -- start start 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bf4a95700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 0x7f9bf010d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bf4a95700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf01000c0 0x7f9bf010dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bf4a95700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf010e570 con 0x7f9bf00663e0 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bf4a95700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf010edb0 con 0x7f9bf01000c0 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 0x7f9bf010d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 0x7f9bf010d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45382/0 (socket says 172.21.15.55:45382) 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 -- 172.21.15.55:0/242710329 learned_addr learned my addr 172.21.15.55:0/242710329 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bedd9b700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf01000c0 0x7f9bf010dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 -- 172.21.15.55:0/242710329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf01000c0 msgr2=0x7f9bf010dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:33.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf01000c0 0x7f9bf010dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 -- 172.21.15.55:0/242710329 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4005040 con 0x7f9bf00663e0 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bee59c700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 0x7f9bf010d9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9be000b790 tx=0x7f9be0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9be000f770 con 0x7f9bf00663e0 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9be000fda0 con 0x7f9bf00663e0 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9be000d310 con 0x7f9bf00663e0 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.641+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf010f090 con 0x7f9bf00663e0 2024-02-20T07:54:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.642+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf010f580 con 0x7f9bf00663e0 2024-02-20T07:54:33.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.643+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf004bed0 con 0x7f9bf00663e0 2024-02-20T07:54:33.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.643+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f9be0010420 con 0x7f9bf00663e0 2024-02-20T07:54:33.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.643+0000 7f9bdeffd700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9bd8069210 0x7f9bd806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:33.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.644+0000 7f9bedd9b700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9bd8069210 0x7f9bd806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:33.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.644+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f9be00864b0 con 0x7f9bf00663e0 2024-02-20T07:54:33.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.644+0000 7f9bedd9b700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9bd8069210 0x7f9bd806b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9be400b3b0 tx=0x7f9be4009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:33.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.648+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9be0056f40 con 0x7f9bf00663e0 2024-02-20T07:54:33.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:33.849+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bf0100d60 con 0x7f9bd8069210 2024-02-20T07:54:34.069 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:34 smithi055 ceph-mon[29254]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:34.069 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:34 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:54:34.069 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:34 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:54:34.069 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:34 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:34.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:34 smithi186 ceph-mon[36772]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:34.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:34 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:54:34.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:34 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:54:34.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:34 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:35.275 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:35 smithi055 ceph-mon[29254]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:35.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:35 smithi186 ceph-mon[36772]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:36.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:36 smithi055 ceph-mon[29254]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:36.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:36 smithi186 ceph-mon[36772]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: Detected new or changed devices on smithi055 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: Adjusting osd_memory_target on smithi055 to 6434M 2024-02-20T07:54:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:38.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:38.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:38.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:37 smithi055 ceph-mon[29254]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: Detected new or changed devices on smithi055 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: Adjusting osd_memory_target on smithi055 to 6434M 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:37 smithi186 ceph-mon[36772]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:39.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:39 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1570900368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecf84045-e8d4-4ef5-bbdb-c95d21ce4647"}]: dispatch 2024-02-20T07:54:39.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:39 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1570900368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecf84045-e8d4-4ef5-bbdb-c95d21ce4647"}]': finished 2024-02-20T07:54:39.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:39 smithi055 ceph-mon[29254]: osdmap e16: 3 total, 2 up, 3 in 2024-02-20T07:54:39.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:39 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:39.417 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:39 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3339101860' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:54:39.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:39 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1570900368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecf84045-e8d4-4ef5-bbdb-c95d21ce4647"}]: dispatch 2024-02-20T07:54:39.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:39 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1570900368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecf84045-e8d4-4ef5-bbdb-c95d21ce4647"}]': finished 2024-02-20T07:54:39.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:39 smithi186 ceph-mon[36772]: osdmap e16: 3 total, 2 up, 3 in 2024-02-20T07:54:39.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:39 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:39.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:39 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3339101860' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:54:40.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:40 smithi055 ceph-mon[29254]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:40.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:40 smithi186 ceph-mon[36772]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:42 smithi186 ceph-mon[36772]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:42 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:42 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:42 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:42 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:42.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:42 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:42.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:42 smithi055 ceph-mon[29254]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:42.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:42 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:42.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:42 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:42.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:42 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:42.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:42 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:42.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:42 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:44.296 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:44 smithi055 ceph-mon[29254]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:44.296 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:44.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:44 smithi186 ceph-mon[36772]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:46.542 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:46 smithi055 ceph-mon[29254]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:46.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:46 smithi186 ceph-mon[36772]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-20T07:54:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:47 smithi055 ceph-mon[29254]: Deploying daemon osd.2 on smithi055 2024-02-20T07:54:47.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-20T07:54:47.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:47.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:47 smithi186 ceph-mon[36772]: Deploying daemon osd.2 on smithi055 2024-02-20T07:54:48.510 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:48 smithi055 ceph-mon[29254]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:48.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:48 smithi186 ceph-mon[36772]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:50.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:50 smithi055 ceph-mon[29254]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:50.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:50 smithi186 ceph-mon[36772]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:52.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:52 smithi055 ceph-mon[29254]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:52.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:52 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:52.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:52 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:52.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:52 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:52.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:52 smithi186 ceph-mon[36772]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:52.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:52 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:52.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:52 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:52.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:52 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:54.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:54 smithi186 ceph-mon[36772]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:54.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:54 smithi055 ceph-mon[29254]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:54.918 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 2 on host 'smithi055' 2024-02-20T07:54:54.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.915+0000 7f9bdeffd700 1 -- 172.21.15.55:0/242710329 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9bf0100d60 con 0x7f9bd8069210 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.918+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9bd8069210 msgr2=0x7f9bd806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.918+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9bd8069210 0x7f9bd806b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9be400b3b0 tx=0x7f9be4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.918+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 msgr2=0x7f9bf010d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.918+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 0x7f9bf010d9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9be000b790 tx=0x7f9be0009250 comp rx=0 tx=0).stop 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 shutdown_connections 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9bd8069210 0x7f9bd806b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9bf00663e0 0x7f9bf010d9a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 --2- 172.21.15.55:0/242710329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf01000c0 0x7f9bf010dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 >> 172.21.15.55:0/242710329 conn(0x7f9bf00f74e0 msgr2=0x7f9bf00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 shutdown_connections 2024-02-20T07:54:54.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:54:54.919+0000 7f9bf4a95700 1 -- 172.21.15.55:0/242710329 wait complete. 2024-02-20T07:54:55.612 DEBUG:teuthology.orchestra.run.smithi055:osd.2> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.2.service 2024-02-20T07:54:55.615 INFO:tasks.cephadm:Deploying osd.3 on smithi186 with /dev/vg_nvme/lv_4... 2024-02-20T07:54:55.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-20T07:54:55.871 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:54:55.993 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:55 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:55.993 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:55 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:55.993 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:55 smithi186 ceph-mon[36772]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:55.993 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:55 smithi186 ceph-mon[36772]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-20T07:54:56.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:55 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:56.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:55 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:56.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:55 smithi055 ceph-mon[29254]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:56.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:55 smithi055 ceph-mon[29254]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-20T07:54:57.194 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:56 smithi186 ceph-mon[36772]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-20T07:54:57.194 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:56 smithi186 ceph-mon[36772]: osdmap e17: 3 total, 2 up, 3 in 2024-02-20T07:54:57.194 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:56 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:57.194 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:56 smithi186 ceph-mon[36772]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-02-20T07:54:57.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:56 smithi055 ceph-mon[29254]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-20T07:54:57.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:56 smithi055 ceph-mon[29254]: osdmap e17: 3 total, 2 up, 3 in 2024-02-20T07:54:57.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:56 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:57.211 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:56 smithi055 ceph-mon[29254]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-02-20T07:54:57.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:54:56.908+0000 7f150bd6a700 -1 osd.2 0 waiting for initial osdmap 2024-02-20T07:54:57.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:54:56.912+0000 7f1506364700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:54:57.854 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-20T07:54:57.854 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-20T07:54:57.854 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-02-20T07:54:57.854 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-02-20T07:54:57.854 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114746 s, 914 MB/s 2024-02-20T07:54:57.855 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-02-20T07:54:58.145 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:57 smithi186 ceph-mon[36772]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-02-20T07:54:58.145 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:57 smithi186 ceph-mon[36772]: osdmap e18: 3 total, 2 up, 3 in 2024-02-20T07:54:58.145 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:57 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:58.146 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:57 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:58.146 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:57 smithi186 ceph-mon[36772]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:58.173 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:57 smithi055 ceph-mon[29254]: from='osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-02-20T07:54:58.173 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:57 smithi055 ceph-mon[29254]: osdmap e18: 3 total, 2 up, 3 in 2024-02-20T07:54:58.173 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:57 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:58.173 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:57 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:58.173 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:57 smithi055 ceph-mon[29254]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:54:58.403 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-02-20T07:54:58.675 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:54:58.957 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: purged_snaps scrub starts 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: purged_snaps scrub ok 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] boot 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: osdmap e19: 3 total, 3 up, 3 in 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:58.958 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: purged_snaps scrub starts 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: purged_snaps scrub ok 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: osd.2 [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] boot 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: osdmap e19: 3 total, 3 up, 3 in 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:59.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:54:59.976 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:59 smithi186 ceph-mon[36772]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:54:59.976 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:59 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-20T07:54:59.976 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:59 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-20T07:54:59.976 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:59 smithi186 ceph-mon[36772]: osdmap e20: 3 total, 3 up, 3 in 2024-02-20T07:54:59.976 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:54:59 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-20T07:55:00.197 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 -- 172.21.15.186:0/2690023105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 msgr2=0x7f91440fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:00.197 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 --2- 172.21.15.186:0/2690023105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440fe550 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9140004970 tx=0x7f9140009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:00.197 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 -- 172.21.15.186:0/2690023105 shutdown_connections 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 --2- 172.21.15.186:0/2690023105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440fe550 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 --2- 172.21.15.186:0/2690023105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91440fcef0 0x7f91440fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 -- 172.21.15.186:0/2690023105 >> 172.21.15.186:0/2690023105 conn(0x7f91440f8620 msgr2=0x7f91440faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 -- 172.21.15.186:0/2690023105 shutdown_connections 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.194+0000 7f914c302700 1 -- 172.21.15.186:0/2690023105 wait complete. 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914c302700 1 Processor -- start 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914c302700 1 -- start start 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914c302700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91440fcef0 0x7f9144071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914c302700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914c302700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91440708e0 con 0x7f91440fcef0 2024-02-20T07:55:00.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914c302700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9144070a50 con 0x7f91440fe0e0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914989d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914989d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36064/0 (socket says 172.21.15.186:36064) 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914989d700 1 -- 172.21.15.186:0/1356197919 learned_addr learned my addr 172.21.15.186:0/1356197919 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914a09e700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91440fcef0 0x7f9144071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914989d700 1 -- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91440fcef0 msgr2=0x7f9144071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.195+0000 7f914989d700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91440fcef0 0x7f9144071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f914989d700 1 -- 172.21.15.186:0/1356197919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140005040 con 0x7f91440fe0e0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f914989d700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440702e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9140009360 tx=0x7f9140009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914000e040 con 0x7f91440fe0e0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f914000bce0 con 0x7f91440fe0e0 2024-02-20T07:55:00.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914001c630 con 0x7f91440fe0e0 2024-02-20T07:55:00.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9144070cd0 con 0x7f91440fe0e0 2024-02-20T07:55:00.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.196+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9144115920 con 0x7f91440fe0e0 2024-02-20T07:55:00.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.197+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914404bed0 con 0x7f91440fe0e0 2024-02-20T07:55:00.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.197+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f91400139f0 con 0x7f91440fe0e0 2024-02-20T07:55:00.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.198+0000 7f913affd700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9130069210 0x7f913006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:00.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.198+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f9140087a40 con 0x7f91440fe0e0 2024-02-20T07:55:00.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.198+0000 7f914a09e700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9130069210 0x7f913006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:00.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.198+0000 7f914a09e700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9130069210 0x7f913006b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9144071bf0 tx=0x7f9134009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:00.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.202+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9140089e60 con 0x7f91440fe0e0 2024-02-20T07:55:00.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:59 smithi055 ceph-mon[29254]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:00.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:59 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-20T07:55:00.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:59 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-20T07:55:00.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:59 smithi055 ceph-mon[29254]: osdmap e20: 3 total, 3 up, 3 in 2024-02-20T07:55:00.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:54:59 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-20T07:55:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:00.393+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9144102ac0 con 0x7f9130069210 2024-02-20T07:55:01.049 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:00 smithi186 ceph-mon[36772]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:01.049 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:55:01.049 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:55:01.049 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:01.049 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-20T07:55:01.049 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:00 smithi186 ceph-mon[36772]: osdmap e21: 3 total, 3 up, 3 in 2024-02-20T07:55:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:00 smithi055 ceph-mon[29254]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:55:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:55:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-20T07:55:01.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:00 smithi055 ceph-mon[29254]: osdmap e21: 3 total, 3 up, 3 in 2024-02-20T07:55:01.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67927]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:55:01.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67927]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:01.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67927]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:01.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67927]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:02.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67934]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:55:02.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67934]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:02.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67934]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:02.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:55:01 smithi055 sudo[67934]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:02.215 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:01 smithi055 ceph-mon[29254]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:02.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:01 smithi186 ceph-mon[36772]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[67937]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:55:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[67937]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[67937]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[67937]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:02.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68004]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:55:02.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:02.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68004]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:02.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68004]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:02.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:02.984+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9140054b80 con 0x7f91440fe0e0 2024-02-20T07:55:03.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68007]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:55:03.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:03.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68007]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:03.036 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:02 smithi055 sudo[68007]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: osdmap e22: 3 total, 3 up, 3 in 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: Detected new or changed devices on smithi055 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:03.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: Adjusting osd_memory_target on smithi055 to 4289M 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/3535465483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29312f2b-ee24-4c2b-a517-10b3a1740ca8"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29312f2b-ee24-4c2b-a517-10b3a1740ca8"}]: dispatch 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29312f2b-ee24-4c2b-a517-10b3a1740ca8"}]': finished 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: osdmap e23: 4 total, 3 up, 4 in 2024-02-20T07:55:03.037 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:02 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68010]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:55:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68010]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68010]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68010]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:03.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: osdmap e22: 3 total, 3 up, 3 in 2024-02-20T07:55:03.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: Detected new or changed devices on smithi055 2024-02-20T07:55:03.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: Adjusting osd_memory_target on smithi055 to 4289M 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/3535465483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29312f2b-ee24-4c2b-a517-10b3a1740ca8"}]: dispatch 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29312f2b-ee24-4c2b-a517-10b3a1740ca8"}]: dispatch 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29312f2b-ee24-4c2b-a517-10b3a1740ca8"}]': finished 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: osdmap e23: 4 total, 3 up, 4 in 2024-02-20T07:55:03.327 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:02 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:03.567 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68013]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:55:03.567 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:03.567 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68013]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:03.567 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68013]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68016]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:55:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68016]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68016]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:03 smithi055 sudo[68016]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:04.073 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:03 smithi186 sudo[40252]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:55:04.073 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:03 smithi186 sudo[40252]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:04.073 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:03 smithi186 sudo[40252]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:04.073 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:03 smithi186 sudo[40252]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 sudo[40257]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 sudo[40257]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 sudo[40257]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 sudo[40257]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: mgrmap e19: smithi055.stjper(active, since 2m), standbys: smithi186.shtgck 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/904968368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:55:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:55:04.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:55:04.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:04 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: mgrmap e19: smithi055.stjper(active, since 2m), standbys: smithi186.shtgck 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/904968368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2024-02-20T07:55:04.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:04 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mon metadata", "id": "smithi186"}]: dispatch 2024-02-20T07:55:05.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:05 smithi055 ceph-mon[29254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:55:05.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:05 smithi055 ceph-mon[29254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:55:05.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:05 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:05.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:05 smithi186 ceph-mon[36772]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:55:05.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:05 smithi186 ceph-mon[36772]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:55:05.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:05 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:06.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:06 smithi055 ceph-mon[29254]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:06.573 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:06 smithi186 ceph-mon[36772]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:08.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:08 smithi186 ceph-mon[36772]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:08 smithi055 ceph-mon[29254]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:10.492 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:10 smithi186 ceph-mon[36772]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:10.492 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:10 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-20T07:55:10.492 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:10 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:10.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:10 smithi055 ceph-mon[29254]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:10.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:10 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-20T07:55:10.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:10 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:11.718 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:11 smithi186 ceph-mon[36772]: Deploying daemon osd.3 on smithi186 2024-02-20T07:55:11.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:11 smithi055 ceph-mon[29254]: Deploying daemon osd.3 on smithi186 2024-02-20T07:55:12.552 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:12 smithi186 ceph-mon[36772]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:12.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:12 smithi055 ceph-mon[29254]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:14.275 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:14 smithi186 ceph-mon[36772]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:14.275 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:14 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:14.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:14 smithi055 ceph-mon[29254]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:14.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:14 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:16.192 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:16.192 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:16.192 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:15 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:16.192 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:15 smithi186 ceph-mon[36772]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:16.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:16.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:16.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:15 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:16.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:15 smithi055 ceph-mon[29254]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:18.488 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:18 smithi186 ceph-mon[36772]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:18.560 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 3 on host 'smithi186' 2024-02-20T07:55:18.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.558+0000 7f913affd700 1 -- 172.21.15.186:0/1356197919 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9144102ac0 con 0x7f9130069210 2024-02-20T07:55:18.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9130069210 msgr2=0x7f913006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:18.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9130069210 0x7f913006b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9144071bf0 tx=0x7f9134009040 comp rx=0 tx=0).stop 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 msgr2=0x7f91440702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440702e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9140009360 tx=0x7f9140009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 shutdown_connections 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f9130069210 0x7f913006b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91440fcef0 0x7f9144071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 --2- 172.21.15.186:0/1356197919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91440fe0e0 0x7f91440702e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 >> 172.21.15.186:0/1356197919 conn(0x7f91440f8620 msgr2=0x7f91441013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 shutdown_connections 2024-02-20T07:55:18.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:18.560+0000 7f914c302700 1 -- 172.21.15.186:0/1356197919 wait complete. 2024-02-20T07:55:18.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:18 smithi055 ceph-mon[29254]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:19.149 DEBUG:teuthology.orchestra.run.smithi186:osd.3> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.3.service 2024-02-20T07:55:19.152 INFO:tasks.cephadm:Deploying osd.4 on smithi186 with /dev/vg_nvme/lv_3... 2024-02-20T07:55:19.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-20T07:55:19.452 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:55:19.794 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:19 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.794 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:19 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.795 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:19 smithi186 ceph-mon[36772]: from='osd.3 [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:55:19.795 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:19 smithi186 ceph-mon[36772]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:55:19.795 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:19 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.795 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:19 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:19 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:19 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:19 smithi055 ceph-mon[29254]: from='osd.3 [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:55:19.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:19 smithi055 ceph-mon[29254]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:55:19.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:19 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:19.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:19 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:20.676 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-mon[36772]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:20.677 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-mon[36772]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-20T07:55:20.677 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-mon[36772]: osdmap e24: 4 total, 3 up, 4 in 2024-02-20T07:55:20.677 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:20.677 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-mon[36772]: from='osd.3 [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:20.677 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-mon[36772]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:20.677 INFO:journalctl@ceph.osd.3.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-3[42231]: 2024-02-20T07:55:20.548+0000 7f0672908700 -1 osd.3 0 waiting for initial osdmap 2024-02-20T07:55:20.677 INFO:journalctl@ceph.osd.3.smithi186.stdout:Feb 20 07:55:20 smithi186 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-3[42231]: 2024-02-20T07:55:20.554+0000 7f066b6ff700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:55:20.908 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:20 smithi055 ceph-mon[29254]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:20.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:20 smithi055 ceph-mon[29254]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-20T07:55:20.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:20 smithi055 ceph-mon[29254]: osdmap e24: 4 total, 3 up, 4 in 2024-02-20T07:55:20.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:20 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:20.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:20 smithi055 ceph-mon[29254]: from='osd.3 [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:20.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:20 smithi055 ceph-mon[29254]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:21.274 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-20T07:55:21.274 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-20T07:55:21.274 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-02-20T07:55:21.274 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-02-20T07:55:21.274 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131472 s, 798 MB/s 2024-02-20T07:55:21.274 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-02-20T07:55:21.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:21 smithi186 ceph-mon[36772]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-02-20T07:55:21.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:21 smithi186 ceph-mon[36772]: osdmap e25: 4 total, 3 up, 4 in 2024-02-20T07:55:21.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:21 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:21.815 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:21 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:21.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:21 smithi055 ceph-mon[29254]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-02-20T07:55:21.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:21 smithi055 ceph-mon[29254]: osdmap e25: 4 total, 3 up, 4 in 2024-02-20T07:55:21.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:21 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:21.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:21 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:22.368 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-02-20T07:55:22.781 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: purged_snaps scrub starts 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: purged_snaps scrub ok 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: osd.3 [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] boot 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: osdmap e26: 4 total, 4 up, 4 in 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:22.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:22 smithi186 ceph-mon[36772]: osdmap e27: 4 total, 4 up, 4 in 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: purged_snaps scrub starts 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: purged_snaps scrub ok 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: osd.3 [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] boot 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: osdmap e26: 4 total, 4 up, 4 in 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:55:22.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:22 smithi055 ceph-mon[29254]: osdmap e27: 4 total, 4 up, 4 in 2024-02-20T07:55:24.756 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: osdmap e28: 4 total, 4 up, 4 in 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:24.757 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:24 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 -- 172.21.15.186:0/2545921711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 msgr2=0x7f1f540fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:24.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/2545921711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f540fc250 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1f50004940 tx=0x7f1f50009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:24.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 -- 172.21.15.186:0/2545921711 shutdown_connections 2024-02-20T07:55:24.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/2545921711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f540fcfb0 0x7f1f540fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:24.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/2545921711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f540fc250 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 -- 172.21.15.186:0/2545921711 >> 172.21.15.186:0/2545921711 conn(0x7f1f540f7520 msgr2=0x7f1f540f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 -- 172.21.15.186:0/2545921711 shutdown_connections 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.950+0000 7f1f5c32e700 1 -- 172.21.15.186:0/2545921711 wait complete. 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5c32e700 1 Processor -- start 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5c32e700 1 -- start start 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5c32e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f54110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5c32e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f540fcfb0 0x7f1f54112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5c32e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f54066d10 con 0x7f1f540fcfb0 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5c32e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f54066e50 con 0x7f1f540fbe40 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5a0ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f54110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5a0ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f54110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54762/0 (socket says 172.21.15.186:54762) 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5a0ca700 1 -- 172.21.15.186:0/3080024172 learned_addr learned my addr 172.21.15.186:0/3080024172 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:55:24.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f598c9700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f540fcfb0 0x7f1f54112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.951+0000 7f1f5a0ca700 1 -- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f540fcfb0 msgr2=0x7f1f54112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f5a0ca700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f540fcfb0 0x7f1f54112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f5a0ca700 1 -- 172.21.15.186:0/3080024172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f50005040 con 0x7f1f540fbe40 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f5a0ca700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f54110630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f500093b0 tx=0x7f1f50009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f5000e040 con 0x7f1f540fbe40 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5000bcc0 con 0x7f1f540fbe40 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f54113180 con 0x7f1f540fbe40 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f5001c630 con 0x7f1f540fbe40 2024-02-20T07:55:24.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.952+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f54113670 con 0x7f1f540fbe40 2024-02-20T07:55:24.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.953+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5404bed0 con 0x7f1f540fbe40 2024-02-20T07:55:24.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.953+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1f500139f0 con 0x7f1f540fbe40 2024-02-20T07:55:24.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.954+0000 7f1f4affd700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1f400692a0 0x7f1f4006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:24.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.954+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f1f50087ef0 con 0x7f1f540fbe40 2024-02-20T07:55:24.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.954+0000 7f1f598c9700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1f400692a0 0x7f1f4006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:24.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.954+0000 7f1f598c9700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1f400692a0 0x7f1f4006b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f44004990 tx=0x7f1f44009ee0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:24.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:24.958+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1f500548c0 con 0x7f1f540fbe40 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: osdmap e28: 4 total, 4 up, 4 in 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:24.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:24 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:25.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:25.151+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f54113890 con 0x7f1f400692a0 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: Detected new or changed devices on smithi186 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: Adjusting osd_memory_target on smithi186 to 15936M 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: osdmap e29: 4 total, 4 up, 4 in 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:55:25.821 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:25 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: Detected new or changed devices on smithi186 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: Adjusting osd_memory_target on smithi186 to 15936M 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: osdmap e29: 4 total, 4 up, 4 in 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:55:25.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:25 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:26.768 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:26 smithi186 ceph-mon[36772]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:26.768 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:26 smithi186 ceph-mon[36772]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:26.768 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:26 smithi186 ceph-mon[36772]: osdmap e30: 4 total, 4 up, 4 in 2024-02-20T07:55:26.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:26 smithi055 ceph-mon[29254]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:26.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:26 smithi055 ceph-mon[29254]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:26.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:26 smithi055 ceph-mon[29254]: osdmap e30: 4 total, 4 up, 4 in 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/636003955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c15e91be-1400-4d62-9eb9-1f761108b4ac"}]: dispatch 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c15e91be-1400-4d62-9eb9-1f761108b4ac"}]: dispatch 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c15e91be-1400-4d62-9eb9-1f761108b4ac"}]': finished 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: osdmap e31: 5 total, 4 up, 5 in 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/295925290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:55:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/636003955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c15e91be-1400-4d62-9eb9-1f761108b4ac"}]: dispatch 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c15e91be-1400-4d62-9eb9-1f761108b4ac"}]: dispatch 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c15e91be-1400-4d62-9eb9-1f761108b4ac"}]': finished 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: osdmap e31: 5 total, 4 up, 5 in 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/295925290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:55:29.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:30.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:30 smithi055 ceph-mon[29254]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-20T07:55:31.074 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:30 smithi186 ceph-mon[36772]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-20T07:55:32.927 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:32 smithi186 ceph-mon[36772]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-20T07:55:32.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:32 smithi055 ceph-mon[29254]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-20T07:55:34.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:34 smithi186 ceph-mon[36772]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-20T07:55:34.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:34 smithi055 ceph-mon[29254]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-20T07:55:35.313 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-20T07:55:35.313 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:35.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-20T07:55:35.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:36.308 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:36 smithi186 ceph-mon[36772]: Deploying daemon osd.4 on smithi186 2024-02-20T07:55:36.308 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:36 smithi186 ceph-mon[36772]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:36.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:36 smithi055 ceph-mon[29254]: Deploying daemon osd.4 on smithi186 2024-02-20T07:55:36.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:36 smithi055 ceph-mon[29254]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:38.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:38 smithi055 ceph-mon[29254]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:38.756 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:38 smithi186 ceph-mon[36772]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:40.933 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:40 smithi186 ceph-mon[36772]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:40.934 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:40 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:40.934 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:40 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:40.934 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:40 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:40.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:40 smithi055 ceph-mon[29254]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:40.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:40 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:40.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:40 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:40.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:40 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:42 smithi186 ceph-mon[36772]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:42.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:42 smithi055 ceph-mon[29254]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:43.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.267+0000 7f1f4affd700 1 -- 172.21.15.186:0/3080024172 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1f54113890 con 0x7f1f400692a0 2024-02-20T07:55:43.269 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 4 on host 'smithi186' 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1f400692a0 msgr2=0x7f1f4006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1f400692a0 0x7f1f4006b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f44004990 tx=0x7f1f44009ee0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 msgr2=0x7f1f54110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f54110630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f500093b0 tx=0x7f1f50009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 shutdown_connections 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f540fbe40 0x7f1f54110630 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1f400692a0 0x7f1f4006b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 --2- 172.21.15.186:0/3080024172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f540fcfb0 0x7f1f54112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 >> 172.21.15.186:0/3080024172 conn(0x7f1f540f7520 msgr2=0x7f1f54100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:43.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 shutdown_connections 2024-02-20T07:55:43.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:43.269+0000 7f1f5c32e700 1 -- 172.21.15.186:0/3080024172 wait complete. 2024-02-20T07:55:43.882 DEBUG:teuthology.orchestra.run.smithi186:osd.4> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.4.service 2024-02-20T07:55:43.885 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_2... 2024-02-20T07:55:43.885 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-20T07:55:44.199 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:55:44.552 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.552 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.552 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:44.552 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:44.553 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='osd.4 [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:55:44.553 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:55:44.553 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.553 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='osd.4 [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:44.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:45.503 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-mon[36772]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-20T07:55:45.503 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-mon[36772]: osdmap e32: 5 total, 4 up, 5 in 2024-02-20T07:55:45.504 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:45.504 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-mon[36772]: from='osd.4 [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:45.504 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-mon[36772]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:45.504 INFO:journalctl@ceph.osd.4.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-4[48677]: 2024-02-20T07:55:45.258+0000 7f856331b700 -1 osd.4 0 waiting for initial osdmap 2024-02-20T07:55:45.504 INFO:journalctl@ceph.osd.4.smithi186.stdout:Feb 20 07:55:45 smithi186 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-4[48677]: 2024-02-20T07:55:45.265+0000 7f855d114700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:55:45.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:45 smithi055 ceph-mon[29254]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-20T07:55:45.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:45 smithi055 ceph-mon[29254]: osdmap e32: 5 total, 4 up, 5 in 2024-02-20T07:55:45.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:45.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:45 smithi055 ceph-mon[29254]: from='osd.4 [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:45.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:45 smithi055 ceph-mon[29254]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:55:46.266 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-20T07:55:46.266 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-20T07:55:46.266 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-02-20T07:55:46.266 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-02-20T07:55:46.266 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0211322 s, 496 MB/s 2024-02-20T07:55:46.266 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-02-20T07:55:46.359 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:46 smithi186 ceph-mon[36772]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-02-20T07:55:46.359 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:46 smithi186 ceph-mon[36772]: osdmap e33: 5 total, 4 up, 5 in 2024-02-20T07:55:46.359 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:46.359 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:46.359 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:46 smithi186 ceph-mon[36772]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:46 smithi055 ceph-mon[29254]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-02-20T07:55:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:46 smithi055 ceph-mon[29254]: osdmap e33: 5 total, 4 up, 5 in 2024-02-20T07:55:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:46 smithi055 ceph-mon[29254]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:55:46.859 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-02-20T07:55:47.507 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:47 smithi186 ceph-mon[36772]: purged_snaps scrub starts 2024-02-20T07:55:47.507 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:47 smithi186 ceph-mon[36772]: purged_snaps scrub ok 2024-02-20T07:55:47.507 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:47 smithi186 ceph-mon[36772]: osd.4 [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] boot 2024-02-20T07:55:47.507 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:47 smithi186 ceph-mon[36772]: osdmap e34: 5 total, 5 up, 5 in 2024-02-20T07:55:47.507 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:47 smithi055 ceph-mon[29254]: purged_snaps scrub starts 2024-02-20T07:55:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:47 smithi055 ceph-mon[29254]: purged_snaps scrub ok 2024-02-20T07:55:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:47 smithi055 ceph-mon[29254]: osd.4 [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] boot 2024-02-20T07:55:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:47 smithi055 ceph-mon[29254]: osdmap e34: 5 total, 5 up, 5 in 2024-02-20T07:55:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:55:47.766 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:55:48.400 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:48 smithi186 ceph-mon[36772]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:48.400 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:48 smithi186 ceph-mon[36772]: osdmap e35: 5 total, 5 up, 5 in 2024-02-20T07:55:48.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:48 smithi055 ceph-mon[29254]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:48 smithi055 ceph-mon[29254]: osdmap e35: 5 total, 5 up, 5 in 2024-02-20T07:55:49.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 -- 172.21.15.186:0/3359460368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 msgr2=0x7ff8140fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:49.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/3359460368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff8140fc250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff810009ce0 tx=0x7ff810004d10 comp rx=0 tx=0).stop 2024-02-20T07:55:49.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 -- 172.21.15.186:0/3359460368 shutdown_connections 2024-02-20T07:55:49.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/3359460368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 0x7ff8140fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/3359460368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff8140fc250 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 -- 172.21.15.186:0/3359460368 >> 172.21.15.186:0/3359460368 conn(0x7ff8140f74e0 msgr2=0x7ff8140f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 -- 172.21.15.186:0/3359460368 shutdown_connections 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.842+0000 7ff81c5dd700 1 -- 172.21.15.186:0/3359460368 wait complete. 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 Processor -- start 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 -- start start 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff81410c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 0x7ff81410cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:49.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81410d520 con 0x7ff8140fcfb0 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81410ddc0 con 0x7ff8140fbe40 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff81410c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff81410c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:53602/0 (socket says 172.21.15.186:53602) 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 -- 172.21.15.186:0/923428842 learned_addr learned my addr 172.21.15.186:0/923428842 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff819b78700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 0x7ff81410cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 -- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 msgr2=0x7ff81410cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 0x7ff81410cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 -- 172.21.15.186:0/923428842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff810009940 con 0x7ff8140fbe40 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff819b78700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 0x7ff81410cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81a379700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff81410c9e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff810015040 tx=0x7ff810004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:49.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81000e040 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.843+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81410e040 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.844+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8100128e0 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.844+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81001c840 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.844+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81410e580 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.844+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81404bed0 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.845+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff810012a50 con 0x7ff8140fbe40 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.845+0000 7ff80affd700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8000692a0 0x7ff80006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.845+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7ff810088970 con 0x7ff8140fbe40 2024-02-20T07:55:49.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.846+0000 7ff819b78700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8000692a0 0x7ff80006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:49.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.846+0000 7ff819b78700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8000692a0 0x7ff80006b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff804009fd0 tx=0x7ff804009ee0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:49.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:49.850+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff810058ba0 con 0x7ff8140fbe40 2024-02-20T07:55:50.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:55:50.038+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff81410ebf0 con 0x7ff8000692a0 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: Detected new or changed devices on smithi186 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: Adjusting osd_memory_target on smithi186 to 7968M 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:50.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:50.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:50.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:49 smithi186 ceph-mon[36772]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: Detected new or changed devices on smithi186 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: Adjusting osd_memory_target on smithi186 to 7968M 2024-02-20T07:55:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:55:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:49 smithi055 ceph-mon[29254]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:50 smithi186 ceph-mon[36772]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:55:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:55:51.075 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:51.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:50 smithi055 ceph-mon[29254]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:51.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:55:51.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:55:51.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:52.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:52 smithi186 ceph-mon[36772]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:52.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:52 smithi055 ceph-mon[29254]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:53.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:53 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/2021311061' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a479e8d-266a-489a-821b-962e81638bcf"}]: dispatch 2024-02-20T07:55:53.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:53 smithi186 ceph-mon[36772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a479e8d-266a-489a-821b-962e81638bcf"}]: dispatch 2024-02-20T07:55:53.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:53 smithi186 ceph-mon[36772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a479e8d-266a-489a-821b-962e81638bcf"}]': finished 2024-02-20T07:55:53.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:53 smithi186 ceph-mon[36772]: osdmap e36: 6 total, 5 up, 6 in 2024-02-20T07:55:53.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:53 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:55:53.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:53 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/2021311061' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a479e8d-266a-489a-821b-962e81638bcf"}]: dispatch 2024-02-20T07:55:53.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:53 smithi055 ceph-mon[29254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a479e8d-266a-489a-821b-962e81638bcf"}]: dispatch 2024-02-20T07:55:53.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:53 smithi055 ceph-mon[29254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a479e8d-266a-489a-821b-962e81638bcf"}]': finished 2024-02-20T07:55:53.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:53 smithi055 ceph-mon[29254]: osdmap e36: 6 total, 5 up, 6 in 2024-02-20T07:55:53.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:53 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:55:54.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:54 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/586744581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:55:54.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:54 smithi186 ceph-mon[36772]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:54 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/586744581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:55:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:54 smithi055 ceph-mon[29254]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:56.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:56 smithi186 ceph-mon[36772]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:56.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:56 smithi055 ceph-mon[29254]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:58.475 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:58 smithi186 ceph-mon[36772]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:58.475 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:55:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:58.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:58 smithi055 ceph-mon[29254]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:55:58.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:55:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:00.659 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:00 smithi186 ceph-mon[36772]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:00.659 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-20T07:56:00.659 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:00 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:00.659 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:00 smithi186 ceph-mon[36772]: Deploying daemon osd.5 on smithi186 2024-02-20T07:56:00.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:00 smithi055 ceph-mon[29254]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:00.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-20T07:56:00.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:00 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:00.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:00 smithi055 ceph-mon[29254]: Deploying daemon osd.5 on smithi186 2024-02-20T07:56:02.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:02 smithi186 ceph-mon[36772]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:02.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:02 smithi055 ceph-mon[29254]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:04.074 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:04 smithi186 ceph-mon[36772]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:04.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:04 smithi055 ceph-mon[29254]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:05.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:05 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:05.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:05 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:05.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:05 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:05.735 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:05 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:05.735 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:05 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:05.735 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:05 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:06.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:06 smithi186 ceph-mon[36772]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:06.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:06 smithi055 ceph-mon[29254]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:07.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.197+0000 7ff80affd700 1 -- 172.21.15.186:0/923428842 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff81410ebf0 con 0x7ff8000692a0 2024-02-20T07:56:07.200 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-02-20T07:56:07.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8000692a0 msgr2=0x7ff80006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:07.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8000692a0 0x7ff80006b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff804009fd0 tx=0x7ff804009ee0 comp rx=0 tx=0).stop 2024-02-20T07:56:07.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 msgr2=0x7ff81410c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff81410c9e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff810015040 tx=0x7ff810004d10 comp rx=0 tx=0).stop 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 shutdown_connections 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8140fbe40 0x7ff81410c9e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8000692a0 0x7ff80006b750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 --2- 172.21.15.186:0/923428842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8140fcfb0 0x7ff81410cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.200+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 >> 172.21.15.186:0/923428842 conn(0x7ff8140f74e0 msgr2=0x7ff814100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.201+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 shutdown_connections 2024-02-20T07:56:07.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:07.201+0000 7ff81c5dd700 1 -- 172.21.15.186:0/923428842 wait complete. 2024-02-20T07:56:08.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:08 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:08.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:08 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:08.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:08 smithi055 ceph-mon[29254]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:08.480 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:08 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:08.480 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:08 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:08.480 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:08 smithi186 ceph-mon[36772]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:08.504 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.5.service 2024-02-20T07:56:08.506 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-20T07:56:08.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-20T07:56:08.782 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:09.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:09 smithi055 ceph-mon[29254]: from='osd.5 [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:56:09.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:09 smithi055 ceph-mon[29254]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:56:09.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:09 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:09.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:09 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:09.473 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:09 smithi186 ceph-mon[36772]: from='osd.5 [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:56:09.473 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:09 smithi186 ceph-mon[36772]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:56:09.473 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:09 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:09.473 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:09 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:10.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:10 smithi055 ceph-mon[29254]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-20T07:56:10.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:10 smithi055 ceph-mon[29254]: osdmap e37: 6 total, 5 up, 6 in 2024-02-20T07:56:10.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:10 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:10.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:10 smithi055 ceph-mon[29254]: from='osd.5 [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:56:10.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:10 smithi055 ceph-mon[29254]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:56:10.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:10 smithi055 ceph-mon[29254]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:10.420 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-mon[36772]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-20T07:56:10.420 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-mon[36772]: osdmap e37: 6 total, 5 up, 6 in 2024-02-20T07:56:10.421 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:10.421 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-mon[36772]: from='osd.5 [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:56:10.421 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-mon[36772]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-02-20T07:56:10.421 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-mon[36772]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:56:10.421 INFO:journalctl@ceph.osd.5.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-5[55597]: 2024-02-20T07:56:10.170+0000 7fbe3f27f700 -1 osd.5 0 waiting for initial osdmap 2024-02-20T07:56:10.421 INFO:journalctl@ceph.osd.5.smithi186.stdout:Feb 20 07:56:10 smithi186 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-5[55597]: 2024-02-20T07:56:10.180+0000 7fbe39078700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 -- 172.21.15.55:0/183553990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fbe40 msgr2=0x7facac0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 --2- 172.21.15.55:0/183553990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fbe40 0x7facac0fc250 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009ce0 tx=0x7fac9c004d10 comp rx=0 tx=0).stop 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 -- 172.21.15.55:0/183553990 shutdown_connections 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 --2- 172.21.15.55:0/183553990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facac0fcfb0 0x7facac0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 --2- 172.21.15.55:0/183553990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fbe40 0x7facac0fc250 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 -- 172.21.15.55:0/183553990 >> 172.21.15.55:0/183553990 conn(0x7facac0f7520 msgr2=0x7facac0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 -- 172.21.15.55:0/183553990 shutdown_connections 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 -- 172.21.15.55:0/183553990 wait complete. 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.540+0000 7facb20cc700 1 Processor -- start 2024-02-20T07:56:10.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facb20cc700 1 -- start start 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facb20cc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facac0fbe40 0x7facac10d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facb20cc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 0x7facac10de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facb20cc700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac10e480 con 0x7facac0fcfb0 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facb20cc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac10ed20 con 0x7facac0fbe40 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 0x7facac10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 0x7facac10de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39372/0 (socket says 172.21.15.55:39372) 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 -- 172.21.15.55:0/4195508764 learned_addr learned my addr 172.21.15.55:0/4195508764 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facab7fe700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facac0fbe40 0x7facac10d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 -- 172.21.15.55:0/4195508764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facac0fbe40 msgr2=0x7facac10d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facac0fbe40 0x7facac10d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 -- 172.21.15.55:0/4195508764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c009940 con 0x7facac0fcfb0 2024-02-20T07:56:10.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facaaffd700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 0x7facac10de80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac9400eb40 tx=0x7fac9400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:10.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac94007cc0 con 0x7facac0fcfb0 2024-02-20T07:56:10.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac10efa0 con 0x7facac0fcfb0 2024-02-20T07:56:10.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac9400f040 con 0x7facac0fcfb0 2024-02-20T07:56:10.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.541+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac9400ca10 con 0x7facac0fcfb0 2024-02-20T07:56:10.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.542+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac10f4e0 con 0x7facac0fcfb0 2024-02-20T07:56:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.542+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac04bed0 con 0x7facac0fcfb0 2024-02-20T07:56:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.543+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fac9400cc80 con 0x7facac0fcfb0 2024-02-20T07:56:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.543+0000 7faca3fff700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fac980692a0 0x7fac9806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.543+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fac940869d0 con 0x7facac0fcfb0 2024-02-20T07:56:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.543+0000 7facab7fe700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fac980692a0 0x7fac9806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.544+0000 7facab7fe700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fac980692a0 0x7fac9806b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fac9c015040 tx=0x7fac9c004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:10.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.548+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fac9404dbd0 con 0x7facac0fcfb0 2024-02-20T07:56:10.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.738+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7facac104930 con 0x7facac0fcfb0 2024-02-20T07:56:10.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.738+0000 7faca3fff700 1 -- 172.21.15.55:0/4195508764 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fac94056b70 con 0x7facac0fcfb0 2024-02-20T07:56:10.740 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:56:10.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fac980692a0 msgr2=0x7fac9806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:10.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fac980692a0 0x7fac9806b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fac9c015040 tx=0x7fac9c004d10 comp rx=0 tx=0).stop 2024-02-20T07:56:10.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 msgr2=0x7facac10de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:10.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 0x7facac10de80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac9400eb40 tx=0x7fac9400b040 comp rx=0 tx=0).stop 2024-02-20T07:56:10.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 shutdown_connections 2024-02-20T07:56:10.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facac0fbe40 0x7facac10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:10.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fac980692a0 0x7fac9806b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:10.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 --2- 172.21.15.55:0/4195508764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facac0fcfb0 0x7facac10de80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:10.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 >> 172.21.15.55:0/4195508764 conn(0x7facac0f7520 msgr2=0x7facac065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:10.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 shutdown_connections 2024-02-20T07:56:10.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:10.740+0000 7facb20cc700 1 -- 172.21.15.55:0/4195508764 wait complete. 2024-02-20T07:56:11.462 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:11 smithi055 ceph-mon[29254]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-02-20T07:56:11.462 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:11 smithi055 ceph-mon[29254]: osdmap e38: 6 total, 5 up, 6 in 2024-02-20T07:56:11.462 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:11.463 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:11 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:11.463 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:11 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/4195508764' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:56:11.490 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1708415746,"num_in_osds":6,"osd_in_since":1708415752,"num_remapped_pgs":0} 2024-02-20T07:56:11.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:11 smithi186 ceph-mon[36772]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-02-20T07:56:11.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:11 smithi186 ceph-mon[36772]: osdmap e38: 6 total, 5 up, 6 in 2024-02-20T07:56:11.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:11.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:11 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:11.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:11 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/4195508764' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: purged_snaps scrub starts 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: purged_snaps scrub ok 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: osd.5 [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] boot 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: osdmap e39: 6 total, 6 up, 6 in 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.231 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:12.232 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:56:12.233 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: purged_snaps scrub starts 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: purged_snaps scrub ok 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: osd.5 [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] boot 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: osdmap e39: 6 total, 6 up, 6 in 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:56:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:56:12.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:56:12.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-20T07:56:12.754 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:13.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:13 smithi055 ceph-mon[29254]: Detected new or changed devices on smithi186 2024-02-20T07:56:13.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:13 smithi055 ceph-mon[29254]: Adjusting osd_memory_target on smithi186 to 5312M 2024-02-20T07:56:13.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:13 smithi055 ceph-mon[29254]: osdmap e40: 6 total, 6 up, 6 in 2024-02-20T07:56:13.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:13 smithi186 ceph-mon[36772]: Detected new or changed devices on smithi186 2024-02-20T07:56:13.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:13 smithi186 ceph-mon[36772]: Adjusting osd_memory_target on smithi186 to 5312M 2024-02-20T07:56:13.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:13 smithi186 ceph-mon[36772]: osdmap e40: 6 total, 6 up, 6 in 2024-02-20T07:56:14.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:14 smithi055 ceph-mon[29254]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:14.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:14 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:14.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.498+0000 7f0a48df9700 1 -- 172.21.15.55:0/3476698072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 msgr2=0x7f0a440fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:14.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.498+0000 7f0a48df9700 1 --2- 172.21.15.55:0/3476698072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a440fc250 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0a38004940 tx=0x7f0a38009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- 172.21.15.55:0/3476698072 shutdown_connections 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 --2- 172.21.15.55:0/3476698072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a440fcfb0 0x7f0a440fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 --2- 172.21.15.55:0/3476698072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a440fc250 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- 172.21.15.55:0/3476698072 >> 172.21.15.55:0/3476698072 conn(0x7f0a440f7520 msgr2=0x7f0a440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- 172.21.15.55:0/3476698072 shutdown_connections 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- 172.21.15.55:0/3476698072 wait complete. 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 Processor -- start 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- start start 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a4410d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a440fcfb0 0x7f0a4410dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410e4a0 con 0x7f0a440fbe40 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a48df9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410ed40 con 0x7f0a440fcfb0 2024-02-20T07:56:14.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.499+0000 7f0a4259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a4410d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a4259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a4410d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38828/0 (socket says 172.21.15.55:38828) 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a4259c700 1 -- 172.21.15.55:0/767081557 learned_addr learned my addr 172.21.15.55:0/767081557 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a41d9b700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a440fcfb0 0x7f0a4410dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a4259c700 1 -- 172.21.15.55:0/767081557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a440fcfb0 msgr2=0x7f0a4410dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a4259c700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a440fcfb0 0x7f0a4410dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a4259c700 1 -- 172.21.15.55:0/767081557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38005040 con 0x7f0a440fbe40 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a4259c700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a4410d960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0a380093b0 tx=0x7f0a38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3800e040 con 0x7f0a440fbe40 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a3800bcc0 con 0x7f0a440fbe40 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4410efc0 con 0x7f0a440fbe40 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3801c630 con 0x7f0a440fbe40 2024-02-20T07:56:14.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.500+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4410f500 con 0x7f0a440fbe40 2024-02-20T07:56:14.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.501+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4404bed0 con 0x7f0a440fbe40 2024-02-20T07:56:14.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.502+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0a38013980 con 0x7f0a440fbe40 2024-02-20T07:56:14.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.502+0000 7f0a32ffd700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0a2c0692a0 0x7f0a2c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:14.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.502+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0a38088110 con 0x7f0a440fbe40 2024-02-20T07:56:14.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.502+0000 7f0a41d9b700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0a2c0692a0 0x7f0a2c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:14.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.502+0000 7f0a41d9b700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0a2c0692a0 0x7f0a2c06b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a34004f60 tx=0x7f0a3400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:14.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.507+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0a380582b0 con 0x7f0a440fbe40 2024-02-20T07:56:14.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:14 smithi186 ceph-mon[36772]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:14.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:14 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:14.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.697+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0a4410f900 con 0x7f0a440fbe40 2024-02-20T07:56:14.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.698+0000 7f0a32ffd700 1 -- 172.21.15.55:0/767081557 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f0a38057e40 con 0x7f0a440fbe40 2024-02-20T07:56:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:56:14.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.699+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0a2c0692a0 msgr2=0x7f0a2c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:14.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.699+0000 7f0a48df9700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0a2c0692a0 0x7f0a2c06b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a34004f60 tx=0x7f0a3400b040 comp rx=0 tx=0).stop 2024-02-20T07:56:14.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.699+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 msgr2=0x7f0a4410d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:14.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.699+0000 7f0a48df9700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a4410d960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0a380093b0 tx=0x7f0a38009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:14.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 shutdown_connections 2024-02-20T07:56:14.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0a2c0692a0 0x7f0a2c06b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:14.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a440fbe40 0x7f0a4410d960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:14.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 --2- 172.21.15.55:0/767081557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a440fcfb0 0x7f0a4410dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:14.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 >> 172.21.15.55:0/767081557 conn(0x7f0a440f7520 msgr2=0x7f0a44100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:14.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 shutdown_connections 2024-02-20T07:56:14.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:14.700+0000 7f0a48df9700 1 -- 172.21.15.55:0/767081557 wait complete. 2024-02-20T07:56:15.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:15 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/767081557' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:56:15.401 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:15 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/767081557' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:56:15.426 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1708415771,"num_in_osds":6,"osd_in_since":1708415752,"num_remapped_pgs":0} 2024-02-20T07:56:15.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-20T07:56:15.692 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:16.433 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:16 smithi055 ceph-mon[29254]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:16.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:16 smithi186 ceph-mon[36772]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:17.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 -- 172.21.15.55:0/2870750449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 msgr2=0x7f9620103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 --2- 172.21.15.55:0/2870750449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620103320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9618004940 tx=0x7f9618009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 -- 172.21.15.55:0/2870750449 shutdown_connections 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 --2- 172.21.15.55:0/2870750449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 0x7f96200fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 --2- 172.21.15.55:0/2870750449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620103320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 -- 172.21.15.55:0/2870750449 >> 172.21.15.55:0/2870750449 conn(0x7f96200f8ae0 msgr2=0x7f96200faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 -- 172.21.15.55:0/2870750449 shutdown_connections 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 -- 172.21.15.55:0/2870750449 wait complete. 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.220+0000 7f9627e46700 1 Processor -- start 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9627e46700 1 -- start start 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9627e46700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9627e46700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 0x7f9620113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9627e46700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9620114600 con 0x7f9620102f50 2024-02-20T07:56:17.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9627e46700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962010d930 con 0x7f96201038f0 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38840/0 (socket says 172.21.15.55:38840) 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 -- 172.21.15.55:0/1961379978 learned_addr learned my addr 172.21.15.55:0/1961379978 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f96253e1700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 0x7f9620113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 -- 172.21.15.55:0/1961379978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 msgr2=0x7f9620113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 0x7f9620113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 -- 172.21.15.55:0/1961379978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618005040 con 0x7f9620102f50 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f96253e1700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 0x7f9620113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9625be2700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620113a70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9618009fd0 tx=0x7f9618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f961800e040 con 0x7f9620102f50 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962010dbb0 con 0x7f9620102f50 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f961800bbf0 con 0x7f9620102f50 2024-02-20T07:56:17.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.221+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f961801c3f0 con 0x7f9620102f50 2024-02-20T07:56:17.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.222+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962010e0f0 con 0x7f9620102f50 2024-02-20T07:56:17.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.222+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96200fc6e0 con 0x7f9620102f50 2024-02-20T07:56:17.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.223+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f961800bd60 con 0x7f9620102f50 2024-02-20T07:56:17.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.223+0000 7f961e7fc700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96080692b0 0x7f960806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:17.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.223+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9618088190 con 0x7f9620102f50 2024-02-20T07:56:17.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.223+0000 7f96253e1700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96080692b0 0x7f960806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:17.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.224+0000 7f96253e1700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96080692b0 0x7f960806b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9614007fd0 tx=0x7f9614009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:17.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.228+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9618058290 con 0x7f9620102f50 2024-02-20T07:56:17.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.419+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f962010f050 con 0x7f9620102f50 2024-02-20T07:56:17.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.420+0000 7f961e7fc700 1 -- 172.21.15.55:0/1961379978 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11070 (secure 0 0 0) 0x7f9618057e20 con 0x7f9620102f50 2024-02-20T07:56:17.421 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:56:17.421 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":40,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:50:23.257218+0000","modified":"2024-02-20T07:56:12.172735+0000","last_up_change":"2024-02-20T07:56:11.165108+0000","last_in_change":"2024-02-20T07:55:52.356730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:54:59.573551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"16d58e18-643f-4fc7-bb34-76cb0ba30b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6803","nonce":3812016597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6805","nonce":3812016597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6809","nonce":3812016597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6807","nonce":3812016597}]},"public_addr":"172.21.15.55:6803/3812016597","cluster_addr":"172.21.15.55:6805/3812016597","heartbeat_back_addr":"172.21.15.55:6809/3812016597","heartbeat_front_addr":"172.21.15.55:6807/3812016597","state":["exists","up"]},{"osd":1,"uuid":"089fab11-e47a-425b-83d8-10bc9dce41e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6811","nonce":4284510212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6813","nonce":4284510212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6817","nonce":4284510212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6815","nonce":4284510212}]},"public_addr":"172.21.15.55:6811/4284510212","cluster_addr":"172.21.15.55:6813/4284510212","heartbeat_back_addr":"172.21.15.55:6817/4284510212","heartbeat_front_addr":"172.21.15.55:6815/4284510212","state":["exists","up"]},{"osd":2,"uuid":"ecf84045-e8d4-4ef5-bbdb-c95d21ce4647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6819","nonce":138638858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6821","nonce":138638858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6825","nonce":138638858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6823","nonce":138638858}]},"public_addr":"172.21.15.55:6819/138638858","cluster_addr":"172.21.15.55:6821/138638858","heartbeat_back_addr":"172.21.15.55:6825/138638858","heartbeat_front_addr":"172.21.15.55:6823/138638858","state":["exists","up"]},{"osd":3,"uuid":"29312f2b-ee24-4c2b-a517-10b3a1740ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6801","nonce":3751122696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6803","nonce":3751122696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6807","nonce":3751122696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6805","nonce":3751122696}]},"public_addr":"172.21.15.186:6801/3751122696","cluster_addr":"172.21.15.186:6803/3751122696","heartbeat_back_addr":"172.21.15.186:6807/3751122696","heartbeat_front_addr":"172.21.15.186:6805/3751122696","state":["exists","up"]},{"osd":4,"uuid":"c15e91be-1400-4d62-9eb9-1f761108b4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6809","nonce":710072980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6811","nonce":710072980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6815","nonce":710072980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6813","nonce":710072980}]},"public_addr":"172.21.15.186:6809/710072980","cluster_addr":"172.21.15.186:6811/710072980","heartbeat_back_addr":"172.21.15.186:6815/710072980","heartbeat_front_addr":"172.21.15.186:6813/710072980","state":["exists","up"]},{"osd":5,"uuid":"9a479e8d-266a-489a-821b-962e81638bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6817","nonce":3358744054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6819","nonce":3358744054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6823","nonce":3358744054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6821","nonce":3358744054}]},"public_addr":"172.21.15.186:6817/3358744054","cluster_addr":"172.21.15.186:6819/3358744054","heartbeat_back_addr":"172.21.15.186:6823/3358744054","heartbeat_front_addr":"172.21.15.186:6821/3358744054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:57.185645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:28.096711+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:56.659378+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:19.780229+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:44.427615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:56:09.309844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/2141854148":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2770110765":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/3177458092":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6801/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2074396018":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/1844035838":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/117641079":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/4098881425":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6800/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/2307704898":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/736324230":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6801/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/1896683634":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6801/117641079":"2024-02-21T07:50:53.618922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96080692b0 msgr2=0x7f960806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96080692b0 0x7f960806b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9614007fd0 tx=0x7f9614009040 comp rx=0 tx=0).stop 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 msgr2=0x7f9620113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620113a70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9618009fd0 tx=0x7f9618009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 shutdown_connections 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96080692b0 0x7f960806b760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9620102f50 0x7f9620113a70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:17.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 --2- 172.21.15.55:0/1961379978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96201038f0 0x7f9620113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:17.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 >> 172.21.15.55:0/1961379978 conn(0x7f96200f8ae0 msgr2=0x7f96200fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:17.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 shutdown_connections 2024-02-20T07:56:17.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:17.422+0000 7f9627e46700 1 -- 172.21.15.55:0/1961379978 wait complete. 2024-02-20T07:56:18.161 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-20T07:54:59.573551+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-20T07:56:18.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-20T07:56:18.426 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:18.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:18 smithi055 ceph-mon[29254]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:18.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:18 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1961379978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:56:18.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:18 smithi186 ceph-mon[36772]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:18.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:18 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1961379978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 -- 172.21.15.55:0/791099663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 msgr2=0x7f60b0106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 --2- 172.21.15.55:0/791099663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0106fa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f60a0004940 tx=0x7f60a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 -- 172.21.15.55:0/791099663 shutdown_connections 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 --2- 172.21.15.55:0/791099663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0106fa0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 --2- 172.21.15.55:0/791099663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b00fdfc0 0x7f60b00fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 -- 172.21.15.55:0/791099663 >> 172.21.15.55:0/791099663 conn(0x7f60b00f7a10 msgr2=0x7f60b00f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 -- 172.21.15.55:0/791099663 shutdown_connections 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 -- 172.21.15.55:0/791099663 wait complete. 2024-02-20T07:56:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.271+0000 7f60b83f9700 1 Processor -- start 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b83f9700 1 -- start start 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b83f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b00fdfc0 0x7f60b0113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b83f9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b83f9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0114690 con 0x7f60b00fe960 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b83f9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b010d9c0 con 0x7f60b00fdfc0 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38852/0 (socket says 172.21.15.55:38852) 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 -- 172.21.15.55:0/2562901446 learned_addr learned my addr 172.21.15.55:0/2562901446 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b6195700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b00fdfc0 0x7f60b0113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 -- 172.21.15.55:0/2562901446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b00fdfc0 msgr2=0x7f60b0113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b00fdfc0 0x7f60b0113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 -- 172.21.15.55:0/2562901446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0005040 con 0x7f60b00fe960 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.272+0000 7f60b5994700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0114040 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f60a0004890 tx=0x7f60a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.273+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a000e040 con 0x7f60b00fe960 2024-02-20T07:56:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.273+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b010dca0 con 0x7f60b00fe960 2024-02-20T07:56:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.273+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60a0004290 con 0x7f60b00fe960 2024-02-20T07:56:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.273+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a001c3f0 con 0x7f60b00fe960 2024-02-20T07:56:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.273+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b010e1e0 con 0x7f60b00fe960 2024-02-20T07:56:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.274+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b01045b0 con 0x7f60b00fe960 2024-02-20T07:56:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.274+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f60a000bbf0 con 0x7f60b00fe960 2024-02-20T07:56:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.274+0000 7f60a6ffd700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f609c0692a0 0x7f609c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.275+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f60a0088160 con 0x7f60b00fe960 2024-02-20T07:56:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.275+0000 7f60b6195700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f609c0692a0 0x7f609c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.275+0000 7f60b6195700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f609c0692a0 0x7f609c06b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60ac004f40 tx=0x7f60ac00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:20.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.279+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f60a0058310 con 0x7f60b00fe960 2024-02-20T07:56:20.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.473+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60b00fb610 con 0x7f60b00fe960 2024-02-20T07:56:20.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.473+0000 7f60a6ffd700 1 -- 172.21.15.55:0/2562901446 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60a0057ea0 con 0x7f60b00fe960 2024-02-20T07:56:20.475 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-02-20T07:56:20.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f609c0692a0 msgr2=0x7f609c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:20.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f609c0692a0 0x7f609c06b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60ac004f40 tx=0x7f60ac00b040 comp rx=0 tx=0).stop 2024-02-20T07:56:20.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 msgr2=0x7f60b0114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0114040 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f60a0004890 tx=0x7f60a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 shutdown_connections 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b00fdfc0 0x7f60b0113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f609c0692a0 0x7f609c06b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 --2- 172.21.15.55:0/2562901446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b00fe960 0x7f60b0114040 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 >> 172.21.15.55:0/2562901446 conn(0x7f60b00f7a10 msgr2=0x7f60b0101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 shutdown_connections 2024-02-20T07:56:20.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:20.475+0000 7f60b83f9700 1 -- 172.21.15.55:0/2562901446 wait complete. 2024-02-20T07:56:20.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:20 smithi055 ceph-mon[29254]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:20.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:20 smithi186 ceph-mon[36772]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:21.083 INFO:tasks.cephadm:Setting up client nodes... 2024-02-20T07:56:21.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-20T07:56:21.350 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:21.674 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:21 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2562901446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:56:21.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:21 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2562901446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:56:22.386 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:22 smithi055 ceph-mon[29254]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:22.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:22 smithi186 ceph-mon[36772]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.898+0000 7f127eee9700 1 -- 172.21.15.55:0/3440765293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 msgr2=0x7f12780fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.898+0000 7f127eee9700 1 --2- 172.21.15.55:0/3440765293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f12780fe550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1264004940 tx=0x7f1264009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- 172.21.15.55:0/3440765293 shutdown_connections 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 --2- 172.21.15.55:0/3440765293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f12780fe550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 --2- 172.21.15.55:0/3440765293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12780fcef0 0x7f12780fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- 172.21.15.55:0/3440765293 >> 172.21.15.55:0/3440765293 conn(0x7f12780f8620 msgr2=0x7f12780faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- 172.21.15.55:0/3440765293 shutdown_connections 2024-02-20T07:56:22.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- 172.21.15.55:0/3440765293 wait complete. 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 Processor -- start 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- start start 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12780fcef0 0x7f127810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f127810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127810e4b0 con 0x7f12780fe0e0 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.899+0000 7f127eee9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127810ed50 con 0x7f12780fcef0 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f127810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f127810deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33038/0 (socket says 172.21.15.55:33038) 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 -- 172.21.15.55:0/555553345 learned_addr learned my addr 172.21.15.55:0/555553345 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f127cc85700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12780fcef0 0x7f127810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 -- 172.21.15.55:0/555553345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12780fcef0 msgr2=0x7f127810d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12780fcef0 0x7f127810d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 -- 172.21.15.55:0/555553345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264005040 con 0x7f12780fe0e0 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f1277fff700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f127810deb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f12640093b0 tx=0x7f1264009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126400e040 con 0x7f12780fe0e0 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127810efd0 con 0x7f12780fe0e0 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f126400bcc0 con 0x7f12780fe0e0 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126401c630 con 0x7f12780fe0e0 2024-02-20T07:56:22.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.900+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127810f510 con 0x7f12780fe0e0 2024-02-20T07:56:22.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.901+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127804bed0 con 0x7f12780fe0e0 2024-02-20T07:56:22.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.902+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1264013420 con 0x7f12780fe0e0 2024-02-20T07:56:22.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.902+0000 7f12757fa700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f12680692a0 0x7f126806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:22.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.902+0000 7f127cc85700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f12680692a0 0x7f126806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:22.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.902+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1264088270 con 0x7f12780fe0e0 2024-02-20T07:56:22.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.902+0000 7f127cc85700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f12680692a0 0x7f126806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f126c009fd0 tx=0x7f126c009f00 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:22.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:22.906+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1264058370 con 0x7f12780fe0e0 2024-02-20T07:56:23.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.178+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1278105a30 con 0x7f12780fe0e0 2024-02-20T07:56:23.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.186+0000 7f12757fa700 1 -- 172.21.15.55:0/555553345 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1264057f00 con 0x7f12780fe0e0 2024-02-20T07:56:23.188 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2024-02-20T07:56:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAnW9RlGPC9ChAAZHtwM7hRyzo+/f21n3kVNA== 2024-02-20T07:56:23.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f12680692a0 msgr2=0x7f126806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:23.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f12680692a0 0x7f126806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f126c009fd0 tx=0x7f126c009f00 comp rx=0 tx=0).stop 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 msgr2=0x7f127810deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f127810deb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f12640093b0 tx=0x7f1264009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 shutdown_connections 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12780fcef0 0x7f127810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f12680692a0 0x7f126806b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 --2- 172.21.15.55:0/555553345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12780fe0e0 0x7f127810deb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 >> 172.21.15.55:0/555553345 conn(0x7f12780f8620 msgr2=0x7f12781013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 shutdown_connections 2024-02-20T07:56:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:23.188+0000 7f127eee9700 1 -- 172.21.15.55:0/555553345 wait complete. 2024-02-20T07:56:23.794 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:56:23.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-20T07:56:23.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-20T07:56:23.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-20T07:56:24.105 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi186/config 2024-02-20T07:56:24.226 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:24 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/555553345' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:56:24.226 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:24 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/555553345' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:56:24.226 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:24 smithi186 ceph-mon[36772]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:24.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:24 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/555553345' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:56:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:24 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/555553345' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:56:24.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:24 smithi055 ceph-mon[29254]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:25.602 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 -- 172.21.15.186:0/350453495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fe0e0 msgr2=0x7f0c000fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/350453495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fe0e0 0x7f0c000fe550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0be8004970 tx=0x7f0be8009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 -- 172.21.15.186:0/350453495 shutdown_connections 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/350453495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fe0e0 0x7f0c000fe550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/350453495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c000fcef0 0x7f0c000fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 -- 172.21.15.186:0/350453495 >> 172.21.15.186:0/350453495 conn(0x7f0c000f8620 msgr2=0x7f0c000faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 -- 172.21.15.186:0/350453495 shutdown_connections 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 -- 172.21.15.186:0/350453495 wait complete. 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 Processor -- start 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.600+0000 7f0c05c1f700 1 -- start start 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0c05c1f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 0x7f0c0010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0c05c1f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c000fe0e0 0x7f0c0010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:25.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0c05c1f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0010e430 con 0x7f0c000fe0e0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0c05c1f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0010ecd0 con 0x7f0c000fcef0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 0x7f0c0010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 0x7f0c0010d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41508/0 (socket says 172.21.15.186:41508) 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 -- 172.21.15.186:0/533102401 learned_addr learned my addr 172.21.15.186:0/533102401 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bf6ffd700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c000fe0e0 0x7f0c0010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 -- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c000fe0e0 msgr2=0x7f0c0010de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c000fe0e0 0x7f0c0010de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 -- 172.21.15.186:0/533102401 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8005040 con 0x7f0c000fcef0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bff7fe700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 0x7f0c0010d8f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0002a80 tx=0x7f0bf0004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bf0007cc0 con 0x7f0c000fcef0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0010efb0 con 0x7f0c000fcef0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bf000f040 con 0x7f0c000fcef0 2024-02-20T07:56:25.604 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bf000da30 con 0x7f0c000fcef0 2024-02-20T07:56:25.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.601+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0010f4f0 con 0x7f0c000fcef0 2024-02-20T07:56:25.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.603+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0004bed0 con 0x7f0c000fcef0 2024-02-20T07:56:25.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.603+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0bf000db90 con 0x7f0c000fcef0 2024-02-20T07:56:25.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.603+0000 7f0bfcff9700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0bec0692a0 0x7f0bec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:25.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.603+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0bf0086c00 con 0x7f0c000fcef0 2024-02-20T07:56:25.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.603+0000 7f0bf6ffd700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0bec0692a0 0x7f0bec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:25.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.604+0000 7f0bf6ffd700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0bec0692a0 0x7f0bec06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0be800b800 tx=0x7f0be8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:25.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.608+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0bf008c020 con 0x7f0c000fcef0 2024-02-20T07:56:25.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.859+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0c00105a30 con 0x7f0c000fcef0 2024-02-20T07:56:25.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.869+0000 7f0bfcff9700 1 -- 172.21.15.186:0/533102401 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0bf0056cf0 con 0x7f0c000fcef0 2024-02-20T07:56:25.870 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-02-20T07:56:25.870 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQApW9Rly5ldMxAA58R52pdZcf4HQ/nmgiAKqQ== 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0bec0692a0 msgr2=0x7f0bec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0bec0692a0 0x7f0bec06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0be800b800 tx=0x7f0be8009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 msgr2=0x7f0c0010d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 0x7f0c0010d8f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0002a80 tx=0x7f0bf0004600 comp rx=0 tx=0).stop 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 shutdown_connections 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c000fcef0 0x7f0c0010d8f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:25.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0bec0692a0 0x7f0bec06b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:25.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 --2- 172.21.15.186:0/533102401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c000fe0e0 0x7f0c0010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:25.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 >> 172.21.15.186:0/533102401 conn(0x7f0c000f8620 msgr2=0x7f0c001013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:25.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 shutdown_connections 2024-02-20T07:56:25.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-02-20T07:56:25.871+0000 7f0c05c1f700 1 -- 172.21.15.186:0/533102401 wait complete. 2024-02-20T07:56:26.463 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:56:26.463 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-20T07:56:26.463 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-20T07:56:26.505 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-20T07:56:26.505 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-20T07:56:26.505 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph mgr dump --format=json 2024-02-20T07:56:26.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:26 smithi186 ceph-mon[36772]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:26.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:26 smithi186 ceph-mon[36772]: from='client.? 172.21.15.186:0/533102401' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:56:26.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:26 smithi186 ceph-mon[36772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:56:26.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:26 smithi186 ceph-mon[36772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:56:26.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:26 smithi055 ceph-mon[29254]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:26.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:26 smithi055 ceph-mon[29254]: from='client.? 172.21.15.186:0/533102401' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:56:26.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:26 smithi055 ceph-mon[29254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:56:26.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:26 smithi055 ceph-mon[29254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:56:26.775 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:28.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:28 smithi055 ceph-mon[29254]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:28.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:28.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 -- 172.21.15.55:0/992264404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fb570 msgr2=0x7fa8780fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:28.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 --2- 172.21.15.55:0/992264404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fb570 0x7fa8780fb9e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa868004940 tx=0x7fa868009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:28.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 -- 172.21.15.55:0/992264404 shutdown_connections 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 --2- 172.21.15.55:0/992264404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fb570 0x7fa8780fb9e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 --2- 172.21.15.55:0/992264404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8780fde50 0x7fa8780fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 -- 172.21.15.55:0/992264404 >> 172.21.15.55:0/992264404 conn(0x7fa878071560 msgr2=0x7fa878071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 -- 172.21.15.55:0/992264404 shutdown_connections 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 -- 172.21.15.55:0/992264404 wait complete. 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 Processor -- start 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.498+0000 7fa880287700 1 -- start start 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa880287700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8780fb570 0x7fa878115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa880287700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 0x7fa878116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa880287700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878116770 con 0x7fa8780fde50 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa880287700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87810faa0 con 0x7fa8780fb570 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 0x7fa878116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 0x7fa878116120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33050/0 (socket says 172.21.15.55:33050) 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 -- 172.21.15.55:0/2428867133 learned_addr learned my addr 172.21.15.55:0/2428867133 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:28.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87e023700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8780fb570 0x7fa878115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 -- 172.21.15.55:0/2428867133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8780fb570 msgr2=0x7fa878115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8780fb570 0x7fa878115be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 -- 172.21.15.55:0/2428867133 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868005040 con 0x7fa8780fde50 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa87d822700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 0x7fa878116120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa868000c00 tx=0x7fa868009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa868021070 con 0x7fa8780fde50 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa868013dd0 con 0x7fa8780fde50 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87810fd20 con 0x7fa8780fde50 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa86800e070 con 0x7fa8780fde50 2024-02-20T07:56:28.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.499+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878110200 con 0x7fa8780fde50 2024-02-20T07:56:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.500+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa878073080 con 0x7fa8780fde50 2024-02-20T07:56:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.501+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa86801c400 con 0x7fa8780fde50 2024-02-20T07:56:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.501+0000 7fa86effd700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa8640692b0 0x7fa86406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.501+0000 7fa87e023700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa8640692b0 0x7fa86406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.501+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa8680883a0 con 0x7fa8780fde50 2024-02-20T07:56:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.502+0000 7fa87e023700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa8640692b0 0x7fa86406b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa874004600 tx=0x7fa87400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:28.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.506+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa8680584a0 con 0x7fa8780fde50 2024-02-20T07:56:28.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.754+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa87804bf00 con 0x7fa8780fde50 2024-02-20T07:56:28.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.757+0000 7fa86effd700 1 -- 172.21.15.55:0/2428867133 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167167 (secure 0 0 0) 0x7fa868058030 con 0x7fa8780fde50 2024-02-20T07:56:28.761 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:56:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.759+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa8640692b0 msgr2=0x7fa86406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.759+0000 7fa880287700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa8640692b0 0x7fa86406b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa874004600 tx=0x7fa87400b040 comp rx=0 tx=0).stop 2024-02-20T07:56:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.759+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 msgr2=0x7fa878116120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.759+0000 7fa880287700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 0x7fa878116120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa868000c00 tx=0x7fa868009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 shutdown_connections 2024-02-20T07:56:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8780fb570 0x7fa878115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa8640692b0 0x7fa86406b760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 --2- 172.21.15.55:0/2428867133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8780fde50 0x7fa878116120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 >> 172.21.15.55:0/2428867133 conn(0x7fa878071560 msgr2=0x7fa8780f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 shutdown_connections 2024-02-20T07:56:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:28.760+0000 7fa880287700 1 -- 172.21.15.55:0/2428867133 wait complete. 2024-02-20T07:56:28.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:28 smithi186 ceph-mon[36772]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:28.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:29.530 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":19,"active_gid":14216,"active_name":"smithi055.stjper","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":762822871},{"type":"v1","addr":"172.21.15.55:6801","nonce":762822871}]},"active_addr":"172.21.15.55:6801/762822871","active_change":"2024-02-20T07:52:43.239175+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14236,"name":"smithi186.shtgck","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.55:8443/","prometheus":"http://172.21.15.55:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":302073653}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2258770328}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":1263148704}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":1006287575}]}]} 2024-02-20T07:56:29.534 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-20T07:56:29.534 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-20T07:56:29.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-20T07:56:29.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:29 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2428867133' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-20T07:56:29.803 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:29.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:29 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2428867133' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-20T07:56:30.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:30 smithi055 ceph-mon[29254]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:30 smithi186 ceph-mon[36772]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:31.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 -- 172.21.15.55:0/3626389065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 msgr2=0x7faa28107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/3626389065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28107050 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa1c004970 tx=0x7faa1c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:31.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 -- 172.21.15.55:0/3626389065 shutdown_connections 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/3626389065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28107050 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/3626389065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 0x7faa280fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 -- 172.21.15.55:0/3626389065 >> 172.21.15.55:0/3626389065 conn(0x7faa28071d90 msgr2=0x7faa28072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 -- 172.21.15.55:0/3626389065 shutdown_connections 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 -- 172.21.15.55:0/3626389065 wait complete. 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 Processor -- start 2024-02-20T07:56:31.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.442+0000 7faa2e3b4700 1 -- start start 2024-02-20T07:56:31.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa2e3b4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 0x7faa28113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa2e3b4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa2e3b4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28114690 con 0x7faa280fe8f0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa2e3b4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2810d930 con 0x7faa280fdfe0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33068/0 (socket says 172.21.15.55:33068) 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 -- 172.21.15.55:0/1806997986 learned_addr learned my addr 172.21.15.55:0/1806997986 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 -- 172.21.15.55:0/1806997986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 msgr2=0x7faa28113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa27fff700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 0x7faa28113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 0x7faa28113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 -- 172.21.15.55:0/1806997986 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c005040 con 0x7faa280fe8f0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa27fff700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 0x7faa28113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa277fe700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28113fb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.443+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa1c00e040 con 0x7faa280fe8f0 2024-02-20T07:56:31.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.444+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2810dbb0 con 0x7faa280fe8f0 2024-02-20T07:56:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.444+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa1c00bce0 con 0x7faa280fe8f0 2024-02-20T07:56:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.444+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa1c01c630 con 0x7faa280fe8f0 2024-02-20T07:56:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.444+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2810e0f0 con 0x7faa280fe8f0 2024-02-20T07:56:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.444+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa281046c0 con 0x7faa280fe8f0 2024-02-20T07:56:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.445+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faa1c0139f0 con 0x7faa280fe8f0 2024-02-20T07:56:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.446+0000 7faa24ff9700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7faa100692a0 0x7faa1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:31.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.446+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faa1c088190 con 0x7faa280fe8f0 2024-02-20T07:56:31.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.446+0000 7faa27fff700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7faa100692a0 0x7faa1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:31.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.446+0000 7faa27fff700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7faa100692a0 0x7faa1006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa18006fd0 tx=0x7faa18008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:31.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.450+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7faa1c058290 con 0x7faa280fe8f0 2024-02-20T07:56:31.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.644+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa280738b0 con 0x7faa280fe8f0 2024-02-20T07:56:31.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.645+0000 7faa24ff9700 1 -- 172.21.15.55:0/1806997986 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11070 (secure 0 0 0) 0x7faa1c057e20 con 0x7faa280fe8f0 2024-02-20T07:56:31.646 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:56:31.646 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":40,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:50:23.257218+0000","modified":"2024-02-20T07:56:12.172735+0000","last_up_change":"2024-02-20T07:56:11.165108+0000","last_in_change":"2024-02-20T07:55:52.356730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:54:59.573551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"16d58e18-643f-4fc7-bb34-76cb0ba30b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6803","nonce":3812016597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6805","nonce":3812016597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6809","nonce":3812016597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6807","nonce":3812016597}]},"public_addr":"172.21.15.55:6803/3812016597","cluster_addr":"172.21.15.55:6805/3812016597","heartbeat_back_addr":"172.21.15.55:6809/3812016597","heartbeat_front_addr":"172.21.15.55:6807/3812016597","state":["exists","up"]},{"osd":1,"uuid":"089fab11-e47a-425b-83d8-10bc9dce41e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6811","nonce":4284510212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6813","nonce":4284510212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6817","nonce":4284510212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6815","nonce":4284510212}]},"public_addr":"172.21.15.55:6811/4284510212","cluster_addr":"172.21.15.55:6813/4284510212","heartbeat_back_addr":"172.21.15.55:6817/4284510212","heartbeat_front_addr":"172.21.15.55:6815/4284510212","state":["exists","up"]},{"osd":2,"uuid":"ecf84045-e8d4-4ef5-bbdb-c95d21ce4647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6819","nonce":138638858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6821","nonce":138638858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6825","nonce":138638858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6823","nonce":138638858}]},"public_addr":"172.21.15.55:6819/138638858","cluster_addr":"172.21.15.55:6821/138638858","heartbeat_back_addr":"172.21.15.55:6825/138638858","heartbeat_front_addr":"172.21.15.55:6823/138638858","state":["exists","up"]},{"osd":3,"uuid":"29312f2b-ee24-4c2b-a517-10b3a1740ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6801","nonce":3751122696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6803","nonce":3751122696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6807","nonce":3751122696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6805","nonce":3751122696}]},"public_addr":"172.21.15.186:6801/3751122696","cluster_addr":"172.21.15.186:6803/3751122696","heartbeat_back_addr":"172.21.15.186:6807/3751122696","heartbeat_front_addr":"172.21.15.186:6805/3751122696","state":["exists","up"]},{"osd":4,"uuid":"c15e91be-1400-4d62-9eb9-1f761108b4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6809","nonce":710072980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6811","nonce":710072980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6815","nonce":710072980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6813","nonce":710072980}]},"public_addr":"172.21.15.186:6809/710072980","cluster_addr":"172.21.15.186:6811/710072980","heartbeat_back_addr":"172.21.15.186:6815/710072980","heartbeat_front_addr":"172.21.15.186:6813/710072980","state":["exists","up"]},{"osd":5,"uuid":"9a479e8d-266a-489a-821b-962e81638bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6817","nonce":3358744054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6819","nonce":3358744054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6823","nonce":3358744054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6821","nonce":3358744054}]},"public_addr":"172.21.15.186:6817/3358744054","cluster_addr":"172.21.15.186:6819/3358744054","heartbeat_back_addr":"172.21.15.186:6823/3358744054","heartbeat_front_addr":"172.21.15.186:6821/3358744054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:57.185645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:28.096711+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:56.659378+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:19.780229+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:44.427615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:56:09.309844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/2141854148":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2770110765":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/3177458092":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6801/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2074396018":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/1844035838":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/117641079":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/4098881425":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6800/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/2307704898":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/736324230":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6801/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/1896683634":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6801/117641079":"2024-02-21T07:50:53.618922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:56:31.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7faa100692a0 msgr2=0x7faa1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7faa100692a0 0x7faa1006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa18006fd0 tx=0x7faa18008040 comp rx=0 tx=0).stop 2024-02-20T07:56:31.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 msgr2=0x7faa28113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28113fb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:31.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 shutdown_connections 2024-02-20T07:56:31.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa280fdfe0 0x7faa28113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7faa100692a0 0x7faa1006b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 --2- 172.21.15.55:0/1806997986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa280fe8f0 0x7faa28113fb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 >> 172.21.15.55:0/1806997986 conn(0x7faa28071d90 msgr2=0x7faa28101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:31.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 shutdown_connections 2024-02-20T07:56:31.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:31.647+0000 7faa2e3b4700 1 -- 172.21.15.55:0/1806997986 wait complete. 2024-02-20T07:56:32.343 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-20T07:56:32.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-20T07:56:32.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:32 smithi186 ceph-mon[36772]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:32.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:32 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1806997986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:56:32.605 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:32.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:32 smithi055 ceph-mon[29254]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:32.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:32 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1806997986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 -- 172.21.15.55:0/924349943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03400fb570 msgr2=0x7f03400fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 --2- 172.21.15.55:0/924349943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03400fb570 0x7f03400fb9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0330004940 tx=0x7f0330009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 -- 172.21.15.55:0/924349943 shutdown_connections 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 --2- 172.21.15.55:0/924349943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03400fb570 0x7f03400fb9c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 --2- 172.21.15.55:0/924349943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0340103140 0x7f0340103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 -- 172.21.15.55:0/924349943 >> 172.21.15.55:0/924349943 conn(0x7f0340071560 msgr2=0x7f0340071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 -- 172.21.15.55:0/924349943 shutdown_connections 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 -- 172.21.15.55:0/924349943 wait complete. 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 Processor -- start 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.116+0000 7f0347c44700 1 -- start start 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f0347c44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03400fb570 0x7f0340113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:34.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f0347c44700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 0x7f0340114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f0347c44700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03401146b0 con 0x7f0340103140 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f0347c44700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034010d9e0 con 0x7f03400fb570 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 0x7f0340114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 0x7f0340114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54878/0 (socket says 172.21.15.55:54878) 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 -- 172.21.15.55:0/1457296175 learned_addr learned my addr 172.21.15.55:0/1457296175 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03459e0700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03400fb570 0x7f0340113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 -- 172.21.15.55:0/1457296175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03400fb570 msgr2=0x7f0340113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03400fb570 0x7f0340113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 -- 172.21.15.55:0/1457296175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0330005040 con 0x7f0340103140 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03451df700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 0x7f0340114060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f033000b3b0 tx=0x7f0330009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033000e040 con 0x7f0340103140 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f033000bbf0 con 0x7f0340103140 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034010dc60 con 0x7f0340103140 2024-02-20T07:56:34.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033001c3f0 con 0x7f0340103140 2024-02-20T07:56:34.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.117+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034010e1a0 con 0x7f0340103140 2024-02-20T07:56:34.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.118+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03401054d0 con 0x7f0340103140 2024-02-20T07:56:34.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.119+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f033000bd60 con 0x7f0340103140 2024-02-20T07:56:34.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.119+0000 7f03367fc700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f032c0692a0 0x7f032c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:34.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.119+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03300881c0 con 0x7f0340103140 2024-02-20T07:56:34.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.119+0000 7f03459e0700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f032c0692a0 0x7f032c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:34.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.120+0000 7f03459e0700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f032c0692a0 0x7f032c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f033c004f40 tx=0x7f033c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:34.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.124+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f03300582c0 con 0x7f0340103140 2024-02-20T07:56:34.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.315+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f034010f170 con 0x7f0340103140 2024-02-20T07:56:34.316 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:34 smithi055 ceph-mon[29254]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:34.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.315+0000 7f03367fc700 1 -- 172.21.15.55:0/1457296175 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11070 (secure 0 0 0) 0x7f034010f170 con 0x7f0340103140 2024-02-20T07:56:34.317 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:56:34.317 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":40,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:50:23.257218+0000","modified":"2024-02-20T07:56:12.172735+0000","last_up_change":"2024-02-20T07:56:11.165108+0000","last_in_change":"2024-02-20T07:55:52.356730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:54:59.573551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"16d58e18-643f-4fc7-bb34-76cb0ba30b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6803","nonce":3812016597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6805","nonce":3812016597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6809","nonce":3812016597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6807","nonce":3812016597}]},"public_addr":"172.21.15.55:6803/3812016597","cluster_addr":"172.21.15.55:6805/3812016597","heartbeat_back_addr":"172.21.15.55:6809/3812016597","heartbeat_front_addr":"172.21.15.55:6807/3812016597","state":["exists","up"]},{"osd":1,"uuid":"089fab11-e47a-425b-83d8-10bc9dce41e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6811","nonce":4284510212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6813","nonce":4284510212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6817","nonce":4284510212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6815","nonce":4284510212}]},"public_addr":"172.21.15.55:6811/4284510212","cluster_addr":"172.21.15.55:6813/4284510212","heartbeat_back_addr":"172.21.15.55:6817/4284510212","heartbeat_front_addr":"172.21.15.55:6815/4284510212","state":["exists","up"]},{"osd":2,"uuid":"ecf84045-e8d4-4ef5-bbdb-c95d21ce4647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6819","nonce":138638858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6821","nonce":138638858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6825","nonce":138638858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6823","nonce":138638858}]},"public_addr":"172.21.15.55:6819/138638858","cluster_addr":"172.21.15.55:6821/138638858","heartbeat_back_addr":"172.21.15.55:6825/138638858","heartbeat_front_addr":"172.21.15.55:6823/138638858","state":["exists","up"]},{"osd":3,"uuid":"29312f2b-ee24-4c2b-a517-10b3a1740ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6801","nonce":3751122696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6803","nonce":3751122696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6807","nonce":3751122696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6805","nonce":3751122696}]},"public_addr":"172.21.15.186:6801/3751122696","cluster_addr":"172.21.15.186:6803/3751122696","heartbeat_back_addr":"172.21.15.186:6807/3751122696","heartbeat_front_addr":"172.21.15.186:6805/3751122696","state":["exists","up"]},{"osd":4,"uuid":"c15e91be-1400-4d62-9eb9-1f761108b4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6809","nonce":710072980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6811","nonce":710072980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6815","nonce":710072980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6813","nonce":710072980}]},"public_addr":"172.21.15.186:6809/710072980","cluster_addr":"172.21.15.186:6811/710072980","heartbeat_back_addr":"172.21.15.186:6815/710072980","heartbeat_front_addr":"172.21.15.186:6813/710072980","state":["exists","up"]},{"osd":5,"uuid":"9a479e8d-266a-489a-821b-962e81638bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6817","nonce":3358744054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6819","nonce":3358744054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6823","nonce":3358744054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6821","nonce":3358744054}]},"public_addr":"172.21.15.186:6817/3358744054","cluster_addr":"172.21.15.186:6819/3358744054","heartbeat_back_addr":"172.21.15.186:6823/3358744054","heartbeat_front_addr":"172.21.15.186:6821/3358744054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:57.185645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:28.096711+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:56.659378+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:19.780229+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:44.427615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:56:09.309844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/2141854148":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2770110765":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/3177458092":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6801/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2074396018":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/1844035838":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/117641079":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/4098881425":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6800/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/2307704898":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/736324230":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6801/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/1896683634":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6801/117641079":"2024-02-21T07:50:53.618922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f032c0692a0 msgr2=0x7f032c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f032c0692a0 0x7f032c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f033c004f40 tx=0x7f033c00b040 comp rx=0 tx=0).stop 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 msgr2=0x7f0340114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 0x7f0340114060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f033000b3b0 tx=0x7f0330009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 shutdown_connections 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03400fb570 0x7f0340113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f032c0692a0 0x7f032c06b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:34.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 --2- 172.21.15.55:0/1457296175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0340103140 0x7f0340114060 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:34.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 >> 172.21.15.55:0/1457296175 conn(0x7f0340071560 msgr2=0x7f03400f9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:34.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 shutdown_connections 2024-02-20T07:56:34.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:34.318+0000 7f0347c44700 1 -- 172.21.15.55:0/1457296175 wait complete. 2024-02-20T07:56:34.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:34 smithi186 ceph-mon[36772]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:35.016 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-20T07:56:35.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-20T07:56:35.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-20T07:56:35.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-20T07:56:35.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-20T07:56:35.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-20T07:56:35.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:35 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1457296175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:56:35.348 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:35.349 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:35.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:35 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1457296175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:56:36.285 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:36.286 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:36.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:36 smithi186 ceph-mon[36772]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:36.372 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:36 smithi055 ceph-mon[29254]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:38.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.364+0000 7f357f971700 1 -- 172.21.15.55:0/1229953614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 msgr2=0x7f3578101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.364+0000 7f357f971700 1 --2- 172.21.15.55:0/1229953614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578101a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3574004a50 tx=0x7f357400af40 comp rx=0 tx=0).stop 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.364+0000 7f357f971700 1 -- 172.21.15.55:0/1229953614 shutdown_connections 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.364+0000 7f357f971700 1 --2- 172.21.15.55:0/1229953614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780fb6c0 0x7f35780fbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.364+0000 7f357f971700 1 --2- 172.21.15.55:0/1229953614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578101a80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.364+0000 7f357f971700 1 -- 172.21.15.55:0/1229953614 >> 172.21.15.55:0/1229953614 conn(0x7f3578071560 msgr2=0x7f3578071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 -- 172.21.15.55:0/1229953614 shutdown_connections 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 -- 172.21.15.55:0/1229953614 wait complete. 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 Processor -- start 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 -- start start 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780fb6c0 0x7f3578113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3578114670 con 0x7f35781016b0 2024-02-20T07:56:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357f971700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357810d910 con 0x7f35780fb6c0 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357cf0c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357cf0c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54900/0 (socket says 172.21.15.55:54900) 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357cf0c700 1 -- 172.21.15.55:0/76901852 learned_addr learned my addr 172.21.15.55:0/76901852 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.365+0000 7f357d70d700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780fb6c0 0x7f3578113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f357cf0c700 1 -- 172.21.15.55:0/76901852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780fb6c0 msgr2=0x7f3578113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f357cf0c700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780fb6c0 0x7f3578113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f357cf0c700 1 -- 172.21.15.55:0/76901852 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3574005040 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f357cf0c700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578113f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f356800b790 tx=0x7f3568009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f356800f6f0 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f356800fd20 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357810dbf0 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35680103a0 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.366+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357810e130 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.367+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f357810e700 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.367+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3568011400 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f356dffb700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f35640692a0 0x7f356406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3568086b50 con 0x7f35781016b0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f357d70d700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f35640692a0 0x7f356406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f356dffb700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] conn(0x7f356406ef70 0x7f3564071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 --> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3564071a30 con 0x7f356406ef70 2024-02-20T07:56:38.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f356808b220 con 0x7f35781016b0 2024-02-20T07:56:38.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f357df0e700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] conn(0x7f356406ef70 0x7f3564071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f357d70d700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f35640692a0 0x7f356406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f357400afd0 tx=0x7f357400af40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:38.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.368+0000 7f357df0e700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] conn(0x7f356406ef70 0x7f3564071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:38.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.369+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== osd.1 v2:172.21.15.55:6810/4284510212 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3564071a30 con 0x7f356406ef70 2024-02-20T07:56:38.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.384+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 --> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f357805e690 con 0x7f356406ef70 2024-02-20T07:56:38.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.384+0000 7f356dffb700 1 -- 172.21.15.55:0/76901852 <== osd.1 v2:172.21.15.55:6810/4284510212 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f357805e690 con 0x7f356406ef70 2024-02-20T07:56:38.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.384+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] conn(0x7f356406ef70 msgr2=0x7f3564071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.384+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] conn(0x7f356406ef70 0x7f3564071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f35640692a0 msgr2=0x7f356406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f35640692a0 0x7f356406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f357400afd0 tx=0x7f357400af40 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 msgr2=0x7f3578113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578113f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f356800b790 tx=0x7f3568009250 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 shutdown_connections 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6810/4284510212,v1:172.21.15.55:6811/4284510212] conn(0x7f356406ef70 0x7f3564071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780fb6c0 0x7f3578113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f35640692a0 0x7f356406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 --2- 172.21.15.55:0/76901852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35781016b0 0x7f3578113f90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 >> 172.21.15.55:0/76901852 conn(0x7f3578071560 msgr2=0x7f35780fa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 shutdown_connections 2024-02-20T07:56:38.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.385+0000 7f357f971700 1 -- 172.21.15.55:0/76901852 wait complete. 2024-02-20T07:56:38.452 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:38 smithi055 ceph-mon[29254]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 -- 172.21.15.55:0/1078409461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2600fb560 msgr2=0x7fa2600fb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 --2- 172.21.15.55:0/1078409461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2600fb560 0x7fa2600fb9d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa248004940 tx=0x7fa248009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 -- 172.21.15.55:0/1078409461 shutdown_connections 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 --2- 172.21.15.55:0/1078409461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2600fb560 0x7fa2600fb9d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 --2- 172.21.15.55:0/1078409461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa260102fe0 0x7fa2601033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 -- 172.21.15.55:0/1078409461 >> 172.21.15.55:0/1078409461 conn(0x7fa260071500 msgr2=0x7fa260071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 -- 172.21.15.55:0/1078409461 shutdown_connections 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 -- 172.21.15.55:0/1078409461 wait complete. 2024-02-20T07:56:38.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 Processor -- start 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.548+0000 7fa265375700 1 -- start start 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa265375700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2600fb560 0x7fa260113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa265375700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 0x7fa260114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa265375700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa260114690 con 0x7fa260102fe0 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa265375700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26010d9c0 con 0x7fa2600fb560 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 0x7fa260114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 0x7fa260114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54928/0 (socket says 172.21.15.55:54928) 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 -- 172.21.15.55:0/3181436461 learned_addr learned my addr 172.21.15.55:0/3181436461 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25effd700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2600fb560 0x7fa260113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 -- 172.21.15.55:0/3181436461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2600fb560 msgr2=0x7fa260113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2600fb560 0x7fa260113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 -- 172.21.15.55:0/3181436461 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248005040 con 0x7fa260102fe0 2024-02-20T07:56:38.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa25e7fc700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 0x7fa260114040 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b3b0 tx=0x7fa248009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24800e040 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa24800bbf0 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26010dc40 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24801c3f0 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.549+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26010e180 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.550+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa26010e670 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa24800bd60 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa2577fe700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa24c0692a0 0x7fa24c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa248088b20 con 0x7fa260102fe0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa25effd700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa24c0692a0 0x7fa24c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa2577fe700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] conn(0x7fa24c06ef70 0x7fa24c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:38.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 --> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa24c071a30 con 0x7fa24c06ef70 2024-02-20T07:56:38.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa248088f20 con 0x7fa260102fe0 2024-02-20T07:56:38.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.551+0000 7fa25f7fe700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] conn(0x7fa24c06ef70 0x7fa24c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:38.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.552+0000 7fa25effd700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa24c0692a0 0x7fa24c06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2600658f0 tx=0x7fa25800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:38.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.552+0000 7fa25f7fe700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] conn(0x7fa24c06ef70 0x7fa24c071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:38.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.552+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== osd.0 v2:172.21.15.55:6802/3812016597 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa24c071a30 con 0x7fa24c06ef70 2024-02-20T07:56:38.567 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:38.567 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:38.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.570+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 --> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa26005e610 con 0x7fa24c06ef70 2024-02-20T07:56:38.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.570+0000 7fa2577fe700 1 -- 172.21.15.55:0/3181436461 <== osd.0 v2:172.21.15.55:6802/3812016597 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa26005e610 con 0x7fa24c06ef70 2024-02-20T07:56:38.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] conn(0x7fa24c06ef70 msgr2=0x7fa24c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] conn(0x7fa24c06ef70 0x7fa24c071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa24c0692a0 msgr2=0x7fa24c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa24c0692a0 0x7fa24c06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2600658f0 tx=0x7fa25800b040 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 msgr2=0x7fa260114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 0x7fa260114040 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b3b0 tx=0x7fa248009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 shutdown_connections 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6802/3812016597,v1:172.21.15.55:6803/3812016597] conn(0x7fa24c06ef70 0x7fa24c071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2600fb560 0x7fa260113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fa24c0692a0 0x7fa24c06b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 --2- 172.21.15.55:0/3181436461 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa260102fe0 0x7fa260114040 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 >> 172.21.15.55:0/3181436461 conn(0x7fa260071500 msgr2=0x7fa2600fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 shutdown_connections 2024-02-20T07:56:38.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:38.571+0000 7fa265375700 1 -- 172.21.15.55:0/3181436461 wait complete. 2024-02-20T07:56:38.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:38 smithi186 ceph-mon[36772]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:39.391 INFO:teuthology.orchestra.run.smithi055.stdout:38654705699 2024-02-20T07:56:39.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-20T07:56:39.810 INFO:teuthology.orchestra.run.smithi055.stdout:60129542173 2024-02-20T07:56:39.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-20T07:56:40.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:40 smithi055 ceph-mon[29254]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:40.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:40 smithi186 ceph-mon[36772]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:42.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:42 smithi055 ceph-mon[29254]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:42.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:42 smithi186 ceph-mon[36772]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:44.171 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:44 smithi055 ceph-mon[29254]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:44.171 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:44.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:44 smithi186 ceph-mon[36772]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 -- 172.21.15.55:0/3575286657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164102800 msgr2=0x7f61640fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 --2- 172.21.15.55:0/3575286657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164102800 0x7f61640fbe70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6158004940 tx=0x7f6158009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 -- 172.21.15.55:0/3575286657 shutdown_connections 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 --2- 172.21.15.55:0/3575286657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164102800 0x7f61640fbe70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 --2- 172.21.15.55:0/3575286657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6164101e60 0x7f6164102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 -- 172.21.15.55:0/3575286657 >> 172.21.15.55:0/3575286657 conn(0x7f61640f7a10 msgr2=0x7f61640f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 -- 172.21.15.55:0/3575286657 shutdown_connections 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.396+0000 7f616a908700 1 -- 172.21.15.55:0/3575286657 wait complete. 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f616a908700 1 Processor -- start 2024-02-20T07:56:44.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f616a908700 1 -- start start 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f616a908700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 0x7f61641129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f616a908700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6164102800 0x7f6164112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f616a908700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6164113550 con 0x7f6164101e60 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f616a908700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616410c880 con 0x7f6164102800 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f6163fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 0x7f61641129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f6163fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 0x7f61641129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33372/0 (socket says 172.21.15.55:33372) 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.397+0000 7f6163fff700 1 -- 172.21.15.55:0/146046962 learned_addr learned my addr 172.21.15.55:0/146046962 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f61637fe700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6164102800 0x7f6164112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6163fff700 1 -- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6164102800 msgr2=0x7f6164112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6163fff700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6164102800 0x7f6164112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6163fff700 1 -- 172.21.15.55:0/146046962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158005040 con 0x7f6164101e60 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6163fff700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 0x7f61641129c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f615400ed20 tx=0x7f6154004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f615400cd10 con 0x7f6164101e60 2024-02-20T07:56:44.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f615400ce70 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616410cb00 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6154010870 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.398+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616410d010 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.399+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f616410d5b0 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.399+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6154010ae0 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f6160ff9700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f614c0692a0 0x7f614c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6154089710 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f61637fe700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f614c0692a0 0x7f614c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f6160ff9700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] conn(0x7f614c06ef70 0x7f614c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 --> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f614c071a30 con 0x7f614c06ef70 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f615408bd00 con 0x7f6164101e60 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f61637fe700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f614c0692a0 0x7f614c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f616410dbb0 tx=0x7f6158009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.400+0000 7f6168ea5700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] conn(0x7f614c06ef70 0x7f614c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.401+0000 7f6168ea5700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] conn(0x7f614c06ef70 0x7f614c071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:44.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.402+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== osd.5 v2:172.21.15.186:6816/3358744054 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f614c071a30 con 0x7f614c06ef70 2024-02-20T07:56:44.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 --> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61640fb610 con 0x7f614c06ef70 2024-02-20T07:56:44.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f6160ff9700 1 -- 172.21.15.55:0/146046962 <== osd.5 v2:172.21.15.186:6816/3358744054 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f61640fb610 con 0x7f614c06ef70 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] conn(0x7f614c06ef70 msgr2=0x7f614c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] conn(0x7f614c06ef70 0x7f614c071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f614c0692a0 msgr2=0x7f614c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f614c0692a0 0x7f614c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f616410dbb0 tx=0x7f6158009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 msgr2=0x7f61641129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 0x7f61641129c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f615400ed20 tx=0x7f6154004570 comp rx=0 tx=0).stop 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 shutdown_connections 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:6816/3358744054,v1:172.21.15.186:6817/3358744054] conn(0x7f614c06ef70 0x7f614c071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f614c0692a0 0x7f614c06b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164101e60 0x7f61641129c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.417+0000 7f616a908700 1 --2- 172.21.15.55:0/146046962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6164102800 0x7f6164112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:44.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.418+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 >> 172.21.15.55:0/146046962 conn(0x7f61640f7a10 msgr2=0x7f61640f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:44.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.418+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 shutdown_connections 2024-02-20T07:56:44.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:44.418+0000 7f616a908700 1 -- 172.21.15.55:0/146046962 wait complete. 2024-02-20T07:56:45.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 -- 172.21.15.55:0/3806587299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490102800 msgr2=0x7fc4900fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 --2- 172.21.15.55:0/3806587299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490102800 0x7fc4900fbe70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc484004940 tx=0x7fc484009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 -- 172.21.15.55:0/3806587299 shutdown_connections 2024-02-20T07:56:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 --2- 172.21.15.55:0/3806587299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490102800 0x7fc4900fbe70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 --2- 172.21.15.55:0/3806587299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490101e60 0x7fc490102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 -- 172.21.15.55:0/3806587299 >> 172.21.15.55:0/3806587299 conn(0x7fc4900f7a10 msgr2=0x7fc4900f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 -- 172.21.15.55:0/3806587299 shutdown_connections 2024-02-20T07:56:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.123+0000 7fc4956e8700 1 -- 172.21.15.55:0/3806587299 wait complete. 2024-02-20T07:56:45.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc4956e8700 1 Processor -- start 2024-02-20T07:56:45.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc4956e8700 1 -- start start 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc4956e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 0x7fc49010eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc4956e8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490102800 0x7fc49010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc4956e8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49010f610 con 0x7fc490101e60 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc4956e8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49010f780 con 0x7fc490102800 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc48effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 0x7fc49010eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc48effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 0x7fc49010eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33390/0 (socket says 172.21.15.55:33390) 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.124+0000 7fc48effd700 1 -- 172.21.15.55:0/1413429224 learned_addr learned my addr 172.21.15.55:0/1413429224 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc48e7fc700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490102800 0x7fc49010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc48effd700 1 -- 172.21.15.55:0/1413429224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490102800 msgr2=0x7fc49010f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc48effd700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490102800 0x7fc49010f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc48effd700 1 -- 172.21.15.55:0/1413429224 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484005040 con 0x7fc490101e60 2024-02-20T07:56:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc48e7fc700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490102800 0x7fc49010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:56:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc48effd700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 0x7fc49010eb40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c830 tx=0x7fc480004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc48000a760 con 0x7fc490101e60 2024-02-20T07:56:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc48000ad90 con 0x7fc490101e60 2024-02-20T07:56:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc490117b10 con 0x7fc490101e60 2024-02-20T07:56:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4800113a0 con 0x7fc490101e60 2024-02-20T07:56:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.125+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc490117fd0 con 0x7fc490101e60 2024-02-20T07:56:45.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.126+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc49005e690 con 0x7fc490101e60 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.126+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc480012400 con 0x7fc490101e60 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc4777fe700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc4780692a0 0x7fc47806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc480086de0 con 0x7fc490101e60 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc48e7fc700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc4780692a0 0x7fc47806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc4777fe700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] conn(0x7fc47806ef70 0x7fc478071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 --> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc478071a30 con 0x7fc47806ef70 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc480089450 con 0x7fc490101e60 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc48f7fe700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] conn(0x7fc47806ef70 0x7fc478071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.127+0000 7fc48e7fc700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc4780692a0 0x7fc47806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc490066a30 tx=0x7fc484009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.128+0000 7fc48f7fe700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] conn(0x7fc47806ef70 0x7fc478071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:45.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.129+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== osd.2 v2:172.21.15.55:6818/138638858 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc478071a30 con 0x7fc47806ef70 2024-02-20T07:56:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.155+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 --> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc49005eaf0 con 0x7fc47806ef70 2024-02-20T07:56:45.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4777fe700 1 -- 172.21.15.55:0/1413429224 <== osd.2 v2:172.21.15.55:6818/138638858 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc49005eaf0 con 0x7fc47806ef70 2024-02-20T07:56:45.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] conn(0x7fc47806ef70 msgr2=0x7fc478071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] conn(0x7fc47806ef70 0x7fc478071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc4780692a0 msgr2=0x7fc47806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc4780692a0 0x7fc47806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc490066a30 tx=0x7fc484009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 msgr2=0x7fc49010eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 0x7fc49010eb40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c830 tx=0x7fc480004470 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 shutdown_connections 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6818/138638858,v1:172.21.15.55:6819/138638858] conn(0x7fc47806ef70 0x7fc478071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc4780692a0 0x7fc47806b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc490101e60 0x7fc49010eb40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 --2- 172.21.15.55:0/1413429224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc490102800 0x7fc49010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 >> 172.21.15.55:0/1413429224 conn(0x7fc4900f7a10 msgr2=0x7fc4900f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 shutdown_connections 2024-02-20T07:56:45.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.156+0000 7fc4956e8700 1 -- 172.21.15.55:0/1413429224 wait complete. 2024-02-20T07:56:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.313+0000 7f2eb1d62700 1 -- 172.21.15.55:0/1791052492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 msgr2=0x7f2eac109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.313+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/1791052492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac109440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2e94004940 tx=0x7f2e94009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 -- 172.21.15.55:0/1791052492 shutdown_connections 2024-02-20T07:56:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/1791052492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac109440 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/1791052492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 0x7f2eac0fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 -- 172.21.15.55:0/1791052492 >> 172.21.15.55:0/1791052492 conn(0x7f2eac071d90 msgr2=0x7f2eac072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:45.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 -- 172.21.15.55:0/1791052492 shutdown_connections 2024-02-20T07:56:45.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 -- 172.21.15.55:0/1791052492 wait complete. 2024-02-20T07:56:45.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 Processor -- start 2024-02-20T07:56:45.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 -- start start 2024-02-20T07:56:45.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.314+0000 7f2eb1d62700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 0x7f2eac115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eb1d62700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eb1d62700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac1168d0 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eb1d62700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac10fbe0 con 0x7f2eac0fbdd0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac116280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33400/0 (socket says 172.21.15.55:33400) 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 -- 172.21.15.55:0/3714768659 learned_addr learned my addr 172.21.15.55:0/3714768659 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eab7fe700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 0x7f2eac115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 -- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 msgr2=0x7f2eac115d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 0x7f2eac115d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 -- 172.21.15.55:0/3714768659 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e94005040 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eab7fe700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 0x7f2eac115d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eaaffd700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac116280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2e94000c00 tx=0x7f2e94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9400e040 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e9400bb90 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eac10fe60 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac1103a0 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.315+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9401c630 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.316+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e8c001000 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.317+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2e9400bd00 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.317+0000 7f2ea3fff700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f2e980692b0 0x7f2e9806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2e94088bb0 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2eab7fe700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f2e980692b0 0x7f2e9806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2ea3fff700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] conn(0x7f2e9806ef80 0x7f2e98071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 --> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e98071a40 con 0x7f2e9806ef80 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e94088f90 con 0x7f2eac0fc6e0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2eab7fe700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f2e980692b0 0x7f2e9806b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4001250 tx=0x7f2ea400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2eabfff700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] conn(0x7f2e9806ef80 0x7f2e98071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:45.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.318+0000 7f2eabfff700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] conn(0x7f2e9806ef80 0x7f2e98071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:45.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.319+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== osd.4 v2:172.21.15.186:6808/710072980 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2e98071a40 con 0x7f2e9806ef80 2024-02-20T07:56:45.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.334+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 --> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e8c002d60 con 0x7f2e9806ef80 2024-02-20T07:56:45.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.334+0000 7f2ea3fff700 1 -- 172.21.15.55:0/3714768659 <== osd.4 v2:172.21.15.186:6808/710072980 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2e8c002d60 con 0x7f2e9806ef80 2024-02-20T07:56:45.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] conn(0x7f2e9806ef80 msgr2=0x7f2e98071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] conn(0x7f2e9806ef80 0x7f2e98071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f2e980692b0 msgr2=0x7f2e9806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f2e980692b0 0x7f2e9806b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4001250 tx=0x7f2ea400b040 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 msgr2=0x7f2eac116280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac116280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2e94000c00 tx=0x7f2e94009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 shutdown_connections 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:6808/710072980,v1:172.21.15.186:6809/710072980] conn(0x7f2e9806ef80 0x7f2e98071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eac0fbdd0 0x7f2eac115d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f2e980692b0 0x7f2e9806b760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 --2- 172.21.15.55:0/3714768659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac0fc6e0 0x7f2eac116280 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 >> 172.21.15.55:0/3714768659 conn(0x7f2eac071d90 msgr2=0x7f2eac0fa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 shutdown_connections 2024-02-20T07:56:45.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:45.335+0000 7f2eb1d62700 1 -- 172.21.15.55:0/3714768659 wait complete. 2024-02-20T07:56:46.032 INFO:teuthology.orchestra.run.smithi055.stdout:81604378649 2024-02-20T07:56:46.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-20T07:56:46.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:46 smithi055 ceph-mon[29254]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:46.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:46 smithi186 ceph-mon[36772]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:47.361 INFO:teuthology.orchestra.run.smithi055.stdout:167503724553 2024-02-20T07:56:47.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-20T07:56:47.994 INFO:teuthology.orchestra.run.smithi055.stdout:146028888078 2024-02-20T07:56:47.994 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-20T07:56:48.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/1348432426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f40fb570 msgr2=0x7ff8f40fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:48.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/1348432426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f40fb570 0x7ff8f40fb9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8004940 tx=0x7ff8e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:48.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/1348432426 shutdown_connections 2024-02-20T07:56:48.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/1348432426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f40fb570 0x7ff8f40fb9c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/1348432426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f4103140 0x7ff8f4103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/1348432426 >> 172.21.15.55:0/1348432426 conn(0x7ff8f4071560 msgr2=0x7ff8f4071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/1348432426 shutdown_connections 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/1348432426 wait complete. 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 Processor -- start 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.182+0000 7ff8f9ae6700 1 -- start start 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f9ae6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f40fb570 0x7ff8f40fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f9ae6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 0x7ff8f40fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f9ae6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40fd290 con 0x7ff8f4103140 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f9ae6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40fd400 con 0x7ff8f40fb570 2024-02-20T07:56:48.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 0x7ff8f40fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 0x7ff8f40fcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33418/0 (socket says 172.21.15.55:33418) 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 -- 172.21.15.55:0/370072355 learned_addr learned my addr 172.21.15.55:0/370072355 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f37fe700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f40fb570 0x7ff8f40fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 -- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f40fb570 msgr2=0x7ff8f40fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f40fb570 0x7ff8f40fc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 -- 172.21.15.55:0/370072355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8005040 con 0x7ff8f4103140 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.183+0000 7ff8f2ffd700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 0x7ff8f40fcc90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009fd0 tx=0x7ff8e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.184+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e800e040 con 0x7ff8f4103140 2024-02-20T07:56:48.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.184+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8e800bbf0 con 0x7ff8f4103140 2024-02-20T07:56:48.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.184+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f4119dd0 con 0x7ff8f4103140 2024-02-20T07:56:48.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.184+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e801c3f0 con 0x7ff8f4103140 2024-02-20T07:56:48.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.184+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f411a230 con 0x7ff8f4103140 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.185+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8f405e610 con 0x7ff8f4103140 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.185+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff8e800bd60 con 0x7ff8f4103140 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8dbfff700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8dc069300 0x7ff8dc06b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff8e8088200 con 0x7ff8f4103140 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8dbfff700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] conn(0x7ff8dc06efd0 0x7ff8dc0713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 --> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8dc071a90 con 0x7ff8dc06efd0 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8e808a7a0 con 0x7ff8f4103140 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8f37fe700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8dc069300 0x7ff8dc06b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8f3fff700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] conn(0x7ff8dc06efd0 0x7ff8dc0713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.186+0000 7ff8f37fe700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8dc069300 0x7ff8dc06b7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4004c40 tx=0x7ff8e4009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.187+0000 7ff8f3fff700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] conn(0x7ff8dc06efd0 0x7ff8dc0713e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.187+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== osd.3 v2:172.21.15.186:6800/3751122696 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff8dc071a90 con 0x7ff8dc06efd0 2024-02-20T07:56:48.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.203+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 --> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff8f405e9c0 con 0x7ff8dc06efd0 2024-02-20T07:56:48.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8dbfff700 1 -- 172.21.15.55:0/370072355 <== osd.3 v2:172.21.15.186:6800/3751122696 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff8f405e9c0 con 0x7ff8dc06efd0 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] conn(0x7ff8dc06efd0 msgr2=0x7ff8dc0713e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] conn(0x7ff8dc06efd0 0x7ff8dc0713e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8dc069300 msgr2=0x7ff8dc06b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8dc069300 0x7ff8dc06b7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4004c40 tx=0x7ff8e4009040 comp rx=0 tx=0).stop 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 msgr2=0x7ff8f40fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 0x7ff8f40fcc90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009fd0 tx=0x7ff8e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 shutdown_connections 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:6800/3751122696,v1:172.21.15.186:6801/3751122696] conn(0x7ff8dc06efd0 0x7ff8dc0713e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f40fb570 0x7ff8f40fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff8dc069300 0x7ff8dc06b7b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 --2- 172.21.15.55:0/370072355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff8f4103140 0x7ff8f40fcc90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.204+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 >> 172.21.15.55:0/370072355 conn(0x7ff8f4071560 msgr2=0x7ff8f40f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.205+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 shutdown_connections 2024-02-20T07:56:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:48.205+0000 7ff8f9ae6700 1 -- 172.21.15.55:0/370072355 wait complete. 2024-02-20T07:56:48.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:48 smithi055 ceph-mon[29254]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:48.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:48 smithi186 ceph-mon[36772]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:49.010 INFO:teuthology.orchestra.run.smithi055.stdout:111669149717 2024-02-20T07:56:49.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-20T07:56:49.034 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:49.082 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:49.086 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:49.088 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:49.094 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:50.461 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:50 smithi055 ceph-mon[29254]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:50.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:50 smithi186 ceph-mon[36772]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:52.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:52 smithi186 ceph-mon[36772]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:52.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:52 smithi055 ceph-mon[29254]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:54.112 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:54 smithi055 ceph-mon[29254]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:54.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:54 smithi186 ceph-mon[36772]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 -- 172.21.15.55:0/1543975181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 msgr2=0x7f58b00706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 --2- 172.21.15.55:0/1543975181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b00706a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f58a0004970 tx=0x7f58a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 -- 172.21.15.55:0/1543975181 shutdown_connections 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 --2- 172.21.15.55:0/1543975181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58b0070be0 0x7f58b01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 --2- 172.21.15.55:0/1543975181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b00706a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 -- 172.21.15.55:0/1543975181 >> 172.21.15.55:0/1543975181 conn(0x7f58b00f8ae0 msgr2=0x7f58b00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 -- 172.21.15.55:0/1543975181 shutdown_connections 2024-02-20T07:56:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 -- 172.21.15.55:0/1543975181 wait complete. 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 Processor -- start 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.330+0000 7f58b6254700 1 -- start start 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58b6254700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b0115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58b6254700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58b0070be0 0x7f58b0116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58b6254700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b01167b0 con 0x7f58b00702d0 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58b6254700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b010fae0 con 0x7f58b0070be0 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b0115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b0115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55248/0 (socket says 172.21.15.55:55248) 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 -- 172.21.15.55:0/740955785 learned_addr learned my addr 172.21.15.55:0/740955785 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 -- 172.21.15.55:0/740955785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58b0070be0 msgr2=0x7f58b0116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58b0070be0 0x7f58b0116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 -- 172.21.15.55:0/740955785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0005040 con 0x7f58b00702d0 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58af7fe700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b0115c20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f58a0000c00 tx=0x7f58a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a000e040 con 0x7f58b00702d0 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58a000bce0 con 0x7f58b00702d0 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b010fd60 con 0x7f58b00702d0 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.331+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a001c630 con 0x7f58b00702d0 2024-02-20T07:56:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.332+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b01102a0 con 0x7f58b00702d0 2024-02-20T07:56:54.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.332+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b0106980 con 0x7f58b00702d0 2024-02-20T07:56:54.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.333+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f58a00139f0 con 0x7f58b00702d0 2024-02-20T07:56:54.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.333+0000 7f5897fff700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f58980692a0 0x7f589806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:54.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.333+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58a00881c0 con 0x7f58b00702d0 2024-02-20T07:56:54.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.333+0000 7f58aeffd700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f58980692a0 0x7f589806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:54.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.334+0000 7f58aeffd700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f58980692a0 0x7f589806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58b0111450 tx=0x7f58a4007d40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:54.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.338+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f58a00582c0 con 0x7f58b00702d0 2024-02-20T07:56:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.514+0000 7fb2a7904700 1 -- 172.21.15.55:0/3146938151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 msgr2=0x7fb2a00fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.514+0000 7fb2a7904700 1 --2- 172.21.15.55:0/3146938151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fbea0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb29c004940 tx=0x7fb29c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.514+0000 7fb2a7904700 1 -- 172.21.15.55:0/3146938151 shutdown_connections 2024-02-20T07:56:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.514+0000 7fb2a7904700 1 --2- 172.21.15.55:0/3146938151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fbea0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.514+0000 7fb2a7904700 1 --2- 172.21.15.55:0/3146938151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a0101e40 0x7fb2a0102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.514+0000 7fb2a7904700 1 -- 172.21.15.55:0/3146938151 >> 172.21.15.55:0/3146938151 conn(0x7fb2a0071d90 msgr2=0x7fb2a0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 -- 172.21.15.55:0/3146938151 shutdown_connections 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 -- 172.21.15.55:0/3146938151 wait complete. 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 Processor -- start 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 -- start start 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a0101e40 0x7fb2a00fdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fe000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00fe600 con 0x7fb2a01027e0 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a7904700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00fe770 con 0x7fb2a0101e40 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a4e9f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fe000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a4e9f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fe000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55274/0 (socket says 172.21.15.55:55274) 2024-02-20T07:56:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.515+0000 7fb2a4e9f700 1 -- 172.21.15.55:0/2678730643 learned_addr learned my addr 172.21.15.55:0/2678730643 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a56a0700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a0101e40 0x7fb2a00fdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a4e9f700 1 -- 172.21.15.55:0/2678730643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a0101e40 msgr2=0x7fb2a00fdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a4e9f700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a0101e40 0x7fb2a00fdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a4e9f700 1 -- 172.21.15.55:0/2678730643 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c005040 con 0x7fb2a01027e0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a4e9f700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fe000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0090d0 tx=0x7fb29c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb29c00e040 con 0x7fb2a01027e0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb29c00bbf0 con 0x7fb2a01027e0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a0119d00 con 0x7fb2a01027e0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb29c01c630 con 0x7fb2a01027e0 2024-02-20T07:56:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.516+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a011a160 con 0x7fb2a01027e0 2024-02-20T07:56:54.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.517+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a00738b0 con 0x7fb2a01027e0 2024-02-20T07:56:54.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.518+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb29c00bd60 con 0x7fb2a01027e0 2024-02-20T07:56:54.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.518+0000 7fb291ffb700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692a0 0x7fb28c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:54.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.518+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb29c0881f0 con 0x7fb2a01027e0 2024-02-20T07:56:54.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.518+0000 7fb2a56a0700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692a0 0x7fb28c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:54.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.518+0000 7fb2a56a0700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692a0 0x7fb28c06b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2940041d0 tx=0x7fb294009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:54.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.522+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb29c054860 con 0x7fb2a01027e0 2024-02-20T07:56:54.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.532+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f58b0111260 con 0x7f58b00702d0 2024-02-20T07:56:54.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.532+0000 7f5897fff700 1 -- 172.21.15.55:0/740955785 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f58a0004de0 con 0x7f58b00702d0 2024-02-20T07:56:54.533 INFO:teuthology.orchestra.run.smithi055.stdout:60129542176 2024-02-20T07:56:54.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f58980692a0 msgr2=0x7f589806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f58980692a0 0x7f589806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58b0111450 tx=0x7f58a4007d40 comp rx=0 tx=0).stop 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 msgr2=0x7f58b0115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b0115c20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f58a0000c00 tx=0x7f58a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 shutdown_connections 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f58980692a0 0x7f589806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58b00702d0 0x7f58b0115c20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 --2- 172.21.15.55:0/740955785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58b0070be0 0x7f58b0116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 >> 172.21.15.55:0/740955785 conn(0x7f58b00f8ae0 msgr2=0x7f58b0103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 shutdown_connections 2024-02-20T07:56:54.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.535+0000 7f58b6254700 1 -- 172.21.15.55:0/740955785 wait complete. 2024-02-20T07:56:54.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.737+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb2a00fe930 con 0x7fb2a01027e0 2024-02-20T07:56:54.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.737+0000 7fb291ffb700 1 -- 172.21.15.55:0/2678730643 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb2a00fe930 con 0x7fb2a01027e0 2024-02-20T07:56:54.739 INFO:teuthology.orchestra.run.smithi055.stdout:38654705702 2024-02-20T07:56:54.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692a0 msgr2=0x7fb28c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692a0 0x7fb28c06b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2940041d0 tx=0x7fb294009040 comp rx=0 tx=0).stop 2024-02-20T07:56:54.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 msgr2=0x7fb2a00fe000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fe000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0090d0 tx=0x7fb29c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 shutdown_connections 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a0101e40 0x7fb2a00fdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692a0 0x7fb28c06b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 --2- 172.21.15.55:0/2678730643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a01027e0 0x7fb2a00fe000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 >> 172.21.15.55:0/2678730643 conn(0x7fb2a0071d90 msgr2=0x7fb2a0065720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 shutdown_connections 2024-02-20T07:56:54.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:54.739+0000 7fb2a7904700 1 -- 172.21.15.55:0/2678730643 wait complete. 2024-02-20T07:56:55.321 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:55 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/740955785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-20T07:56:55.321 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:55 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2678730643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-20T07:56:55.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:55 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/740955785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-20T07:56:55.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:55 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2678730643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-20T07:56:55.345 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-02-20T07:56:55.345 DEBUG:teuthology.parallel:result is None 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 -- 172.21.15.55:0/3465920034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec102f50 msgr2=0x7fdfec103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/3465920034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec102f50 0x7fdfec103320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0004940 tx=0x7fdfe0009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 -- 172.21.15.55:0/3465920034 shutdown_connections 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/3465920034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfec1038f0 0x7fdfec0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/3465920034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec102f50 0x7fdfec103320 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 -- 172.21.15.55:0/3465920034 >> 172.21.15.55:0/3465920034 conn(0x7fdfec0f8ae0 msgr2=0x7fdfec0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 -- 172.21.15.55:0/3465920034 shutdown_connections 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.555+0000 7fdff2bf9700 1 -- 172.21.15.55:0/3465920034 wait complete. 2024-02-20T07:56:55.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdff2bf9700 1 Processor -- start 2024-02-20T07:56:55.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdff2bf9700 1 -- start start 2024-02-20T07:56:55.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdff2bf9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfec102f50 0x7fdfec113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:55.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdff2bf9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 0x7fdfec114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:55.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdff2bf9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec114700 con 0x7fdfec1038f0 2024-02-20T07:56:55.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdff2bf9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec10d9a0 con 0x7fdfec102f50 2024-02-20T07:56:55.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdfebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 0x7fdfec114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdfebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 0x7fdfec114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55286/0 (socket says 172.21.15.55:55286) 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.556+0000 7fdfebfff700 1 -- 172.21.15.55:0/4086887651 learned_addr learned my addr 172.21.15.55:0/4086887651 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdff0995700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfec102f50 0x7fdfec113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfebfff700 1 -- 172.21.15.55:0/4086887651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfec102f50 msgr2=0x7fdfec113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfebfff700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfec102f50 0x7fdfec113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfebfff700 1 -- 172.21.15.55:0/4086887651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe0005040 con 0x7fdfec1038f0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfebfff700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 0x7fdfec114020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800b790 tx=0x7fdfd8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd800f6f0 con 0x7fdfec1038f0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfd800fd20 con 0x7fdfec1038f0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec10dc80 con 0x7fdfec1038f0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd80103a0 con 0x7fdfec1038f0 2024-02-20T07:56:55.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.557+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec10e1c0 con 0x7fdfec1038f0 2024-02-20T07:56:55.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.558+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec0fc6e0 con 0x7fdfec1038f0 2024-02-20T07:56:55.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.559+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdfd8011400 con 0x7fdfec1038f0 2024-02-20T07:56:55.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.559+0000 7fdfe97fa700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fdfdc0692a0 0x7fdfdc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:55.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.559+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdfd8086c20 con 0x7fdfec1038f0 2024-02-20T07:56:55.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.559+0000 7fdff0995700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fdfdc0692a0 0x7fdfdc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:55.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.559+0000 7fdff0995700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fdfdc0692a0 0x7fdfdc06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0004f90 tx=0x7fdfe0009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:55.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.563+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdfd8056d20 con 0x7fdfec1038f0 2024-02-20T07:56:55.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.759+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fdfec04bed0 con 0x7fdfec1038f0 2024-02-20T07:56:55.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.759+0000 7fdfe97fa700 1 -- 172.21.15.55:0/4086887651 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdfd80568b0 con 0x7fdfec1038f0 2024-02-20T07:56:55.761 INFO:teuthology.orchestra.run.smithi055.stdout:146028888080 2024-02-20T07:56:55.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fdfdc0692a0 msgr2=0x7fdfdc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:55.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fdfdc0692a0 0x7fdfdc06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0004f90 tx=0x7fdfe0009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 msgr2=0x7fdfec114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 0x7fdfec114020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800b790 tx=0x7fdfd8009220 comp rx=0 tx=0).stop 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 shutdown_connections 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfec102f50 0x7fdfec113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fdfdc0692a0 0x7fdfdc06b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 --2- 172.21.15.55:0/4086887651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfec1038f0 0x7fdfec114020 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 >> 172.21.15.55:0/4086887651 conn(0x7fdfec0f8ae0 msgr2=0x7fdfec0fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 shutdown_connections 2024-02-20T07:56:55.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:55.761+0000 7fdff2bf9700 1 -- 172.21.15.55:0/4086887651 wait complete. 2024-02-20T07:56:56.300 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:56 smithi055 ceph-mon[29254]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:56.300 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:56 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/4086887651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-20T07:56:56.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:56 smithi186 ceph-mon[36772]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:56.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:56 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/4086887651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-20T07:56:56.840 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-02-20T07:56:56.840 DEBUG:teuthology.parallel:result is None 2024-02-20T07:56:57.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 -- 172.21.15.55:0/845350794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 msgr2=0x7fcc5c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 --2- 172.21.15.55:0/845350794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c102210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c004940 tx=0x7fcc4c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 -- 172.21.15.55:0/845350794 shutdown_connections 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 --2- 172.21.15.55:0/845350794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc5c1027e0 0x7fcc5c0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 --2- 172.21.15.55:0/845350794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c102210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 -- 172.21.15.55:0/845350794 >> 172.21.15.55:0/845350794 conn(0x7fcc5c071d90 msgr2=0x7fcc5c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 -- 172.21.15.55:0/845350794 shutdown_connections 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 -- 172.21.15.55:0/845350794 wait complete. 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.427+0000 7fcc6251d700 1 Processor -- start 2024-02-20T07:56:57.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc6251d700 1 -- start start 2024-02-20T07:56:57.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc6251d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c112970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc6251d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc5c1027e0 0x7fcc5c112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc6251d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c113590 con 0x7fcc5c101e40 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc6251d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c10c830 con 0x7fcc5c1027e0 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c112970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55302/0 (socket says 172.21.15.55:55302) 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 -- 172.21.15.55:0/2818769524 learned_addr learned my addr 172.21.15.55:0/2818769524 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 -- 172.21.15.55:0/2818769524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc5c1027e0 msgr2=0x7fcc5c112eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc5c1027e0 0x7fcc5c112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 -- 172.21.15.55:0/2818769524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c005040 con 0x7fcc5c101e40 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.428+0000 7fcc5bfff700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c112970 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b3b0 tx=0x7fcc4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.429+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc4c00e040 con 0x7fcc5c101e40 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.429+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc4c00bbf0 con 0x7fcc5c101e40 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.429+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c10cab0 con 0x7fcc5c101e40 2024-02-20T07:56:57.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.429+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc4c01c630 con 0x7fcc5c101e40 2024-02-20T07:56:57.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.429+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c10cff0 con 0x7fcc5c101e40 2024-02-20T07:56:57.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.430+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c0738b0 con 0x7fcc5c101e40 2024-02-20T07:56:57.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.430+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcc4c00bd60 con 0x7fcc5c101e40 2024-02-20T07:56:57.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.430+0000 7fcc58ff9700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcc440692a0 0x7fcc4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:57.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.431+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcc4c088170 con 0x7fcc5c101e40 2024-02-20T07:56:57.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.431+0000 7fcc5b7fe700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcc440692a0 0x7fcc4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:57.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.431+0000 7fcc5b7fe700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcc440692a0 0x7fcc4406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcc50006fd0 tx=0x7fcc50006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:57.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.435+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcc4c058310 con 0x7fcc5c101e40 2024-02-20T07:56:57.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.634+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcc5c04bf00 con 0x7fcc5c101e40 2024-02-20T07:56:57.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.634+0000 7fcc58ff9700 1 -- 172.21.15.55:0/2818769524 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc4c057ea0 con 0x7fcc5c101e40 2024-02-20T07:56:57.636 INFO:teuthology.orchestra.run.smithi055.stdout:81604378651 2024-02-20T07:56:57.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcc440692a0 msgr2=0x7fcc4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:57.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcc440692a0 0x7fcc4406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcc50006fd0 tx=0x7fcc50006db0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 msgr2=0x7fcc5c112970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:57.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c112970 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b3b0 tx=0x7fcc4c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:57.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 shutdown_connections 2024-02-20T07:56:57.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcc440692a0 0x7fcc4406b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc5c101e40 0x7fcc5c112970 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 --2- 172.21.15.55:0/2818769524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc5c1027e0 0x7fcc5c112eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:57.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 >> 172.21.15.55:0/2818769524 conn(0x7fcc5c071d90 msgr2=0x7fcc5c0fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:57.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 shutdown_connections 2024-02-20T07:56:57.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:57.636+0000 7fcc6251d700 1 -- 172.21.15.55:0/2818769524 wait complete. 2024-02-20T07:56:58.244 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888080 for osd.4 2024-02-20T07:56:58.244 DEBUG:teuthology.parallel:result is None 2024-02-20T07:56:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:58 smithi055 ceph-mon[29254]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:58.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:58 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2818769524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-20T07:56:58.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:58.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-02-20T07:56:58.725 DEBUG:teuthology.parallel:result is None 2024-02-20T07:56:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:58 smithi186 ceph-mon[36772]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:56:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:58 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2818769524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-20T07:56:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:58.889 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:56:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.930+0000 7fd16570c700 1 -- 172.21.15.55:0/2816554323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd160102f50 msgr2=0x7fd160103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.930+0000 7fd16570c700 1 --2- 172.21.15.55:0/2816554323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd160102f50 0x7fd160103320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd150004940 tx=0x7fd150009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.930+0000 7fd16570c700 1 -- 172.21.15.55:0/2816554323 shutdown_connections 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.930+0000 7fd16570c700 1 --2- 172.21.15.55:0/2816554323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601038f0 0x7fd1600fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.930+0000 7fd16570c700 1 --2- 172.21.15.55:0/2816554323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd160102f50 0x7fd160103320 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.930+0000 7fd16570c700 1 -- 172.21.15.55:0/2816554323 >> 172.21.15.55:0/2816554323 conn(0x7fd1600f8ae0 msgr2=0x7fd1600faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 -- 172.21.15.55:0/2816554323 shutdown_connections 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 -- 172.21.15.55:0/2816554323 wait complete. 2024-02-20T07:56:58.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 Processor -- start 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 -- start start 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd160102f50 0x7fd160113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 0x7fd160114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160114700 con 0x7fd1601038f0 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd16570c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16010d9a0 con 0x7fd160102f50 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.931+0000 7fd157fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 0x7fd160114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd157fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 0x7fd160114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55318/0 (socket says 172.21.15.55:55318) 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd157fff700 1 -- 172.21.15.55:0/575582249 learned_addr learned my addr 172.21.15.55:0/575582249 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:56:58.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd15effd700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd160102f50 0x7fd160113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd157fff700 1 -- 172.21.15.55:0/575582249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd160102f50 msgr2=0x7fd160113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd157fff700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd160102f50 0x7fd160113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd157fff700 1 -- 172.21.15.55:0/575582249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150005040 con 0x7fd1601038f0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd157fff700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 0x7fd160114020 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd14800b790 tx=0x7fd148009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14800f6f0 con 0x7fd1601038f0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16010dc80 con 0x7fd1601038f0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd14800fd20 con 0x7fd1601038f0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1480103a0 con 0x7fd1601038f0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.932+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16010e1c0 con 0x7fd1601038f0 2024-02-20T07:56:58.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.933+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1600fc6e0 con 0x7fd1601038f0 2024-02-20T07:56:58.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.934+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd148011400 con 0x7fd1601038f0 2024-02-20T07:56:58.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.934+0000 7fd1577fe700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd14c0692a0 0x7fd14c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:58.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.934+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd148086d10 con 0x7fd1601038f0 2024-02-20T07:56:58.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.934+0000 7fd15effd700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd14c0692a0 0x7fd14c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:58.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.935+0000 7fd15effd700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd14c0692a0 0x7fd14c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd150004f90 tx=0x7fd150009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:58.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:58.939+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd148056d90 con 0x7fd1601038f0 2024-02-20T07:56:59.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.134+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd16004bed0 con 0x7fd1601038f0 2024-02-20T07:56:59.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.135+0000 7fd1577fe700 1 -- 172.21.15.55:0/575582249 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd148056920 con 0x7fd1601038f0 2024-02-20T07:56:59.136 INFO:teuthology.orchestra.run.smithi055.stdout:167503724556 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd14c0692a0 msgr2=0x7fd14c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd14c0692a0 0x7fd14c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd150004f90 tx=0x7fd150009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 msgr2=0x7fd160114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 0x7fd160114020 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd14800b790 tx=0x7fd148009220 comp rx=0 tx=0).stop 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 shutdown_connections 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd160102f50 0x7fd160113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fd14c0692a0 0x7fd14c06b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 --2- 172.21.15.55:0/575582249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1601038f0 0x7fd160114020 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 >> 172.21.15.55:0/575582249 conn(0x7fd1600f8ae0 msgr2=0x7fd1600fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:59.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 shutdown_connections 2024-02-20T07:56:59.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:56:59.137+0000 7fd16570c700 1 -- 172.21.15.55:0/575582249 wait complete. 2024-02-20T07:56:59.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:56:59 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/575582249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-20T07:56:59.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:56:59 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/575582249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-20T07:57:00.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:00 smithi055 ceph-mon[29254]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:00.576 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724556 for osd.5 2024-02-20T07:57:00.576 DEBUG:teuthology.parallel:result is None 2024-02-20T07:57:00.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:00 smithi186 ceph-mon[36772]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:01.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 -- 172.21.15.55:0/599077275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fdf50 msgr2=0x7f1e140fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:01.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 --2- 172.21.15.55:0/599077275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fdf50 0x7f1e140fe320 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1e08004940 tx=0x7f1e08009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:01.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 -- 172.21.15.55:0/599077275 shutdown_connections 2024-02-20T07:57:01.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 --2- 172.21.15.55:0/599077275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e140fe8f0 0x7f1e14106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:01.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 --2- 172.21.15.55:0/599077275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fdf50 0x7f1e140fe320 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:01.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 -- 172.21.15.55:0/599077275 >> 172.21.15.55:0/599077275 conn(0x7f1e140f7a10 msgr2=0x7f1e140f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 -- 172.21.15.55:0/599077275 shutdown_connections 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.017+0000 7f1e18b28700 1 -- 172.21.15.55:0/599077275 wait complete. 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e18b28700 1 Processor -- start 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e18b28700 1 -- start start 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e18b28700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e140fdf50 0x7f1e14113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e18b28700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 0x7f1e14113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e18b28700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14114660 con 0x7f1e140fe8f0 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e18b28700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1410d900 con 0x7f1e140fdf50 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e11d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 0x7f1e14113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e11d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 0x7f1e14113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55346/0 (socket says 172.21.15.55:55346) 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e11d9b700 1 -- 172.21.15.55:0/2583267066 learned_addr learned my addr 172.21.15.55:0/2583267066 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e1259c700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e140fdf50 0x7f1e14113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e11d9b700 1 -- 172.21.15.55:0/2583267066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e140fdf50 msgr2=0x7f1e14113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e11d9b700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e140fdf50 0x7f1e14113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.018+0000 7f1e11d9b700 1 -- 172.21.15.55:0/2583267066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e08005040 con 0x7f1e140fe8f0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.019+0000 7f1e11d9b700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 0x7f1e14113f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b790 tx=0x7f1e04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.019+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e0400f770 con 0x7f1e140fe8f0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.019+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e0400fda0 con 0x7f1e140fe8f0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.019+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1410dbe0 con 0x7f1e140fe8f0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.019+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e0400d310 con 0x7f1e140fe8f0 2024-02-20T07:57:01.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.019+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1410e120 con 0x7f1e140fe8f0 2024-02-20T07:57:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.020+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e14104620 con 0x7f1e140fe8f0 2024-02-20T07:57:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.020+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1e04010420 con 0x7f1e140fe8f0 2024-02-20T07:57:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.021+0000 7f1e02ffd700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1dfc0692a0 0x7f1dfc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.021+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1e04086d60 con 0x7f1e140fe8f0 2024-02-20T07:57:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.021+0000 7f1e1259c700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1dfc0692a0 0x7f1dfc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.021+0000 7f1e1259c700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1dfc0692a0 0x7f1dfc06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e08004f90 tx=0x7f1e08009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:01.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.025+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1e04056f00 con 0x7f1e140fe8f0 2024-02-20T07:57:01.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.220+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1e140fb610 con 0x7f1e140fe8f0 2024-02-20T07:57:01.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.220+0000 7f1e02ffd700 1 -- 172.21.15.55:0/2583267066 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1e04056a90 con 0x7f1e140fe8f0 2024-02-20T07:57:01.222 INFO:teuthology.orchestra.run.smithi055.stdout:111669149719 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1dfc0692a0 msgr2=0x7f1dfc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1dfc0692a0 0x7f1dfc06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e08004f90 tx=0x7f1e08009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 msgr2=0x7f1e14113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 0x7f1e14113f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b790 tx=0x7f1e04009250 comp rx=0 tx=0).stop 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 shutdown_connections 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e140fdf50 0x7f1e14113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f1dfc0692a0 0x7f1dfc06b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 --2- 172.21.15.55:0/2583267066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e140fe8f0 0x7f1e14113f80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.222+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 >> 172.21.15.55:0/2583267066 conn(0x7f1e140f7a10 msgr2=0x7f1e141017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.223+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 shutdown_connections 2024-02-20T07:57:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:01.223+0000 7f1e18b28700 1 -- 172.21.15.55:0/2583267066 wait complete. 2024-02-20T07:57:01.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:01 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2583267066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-20T07:57:01.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:01 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2583267066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-20T07:57:01.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149717 got 111669149719 for osd.3 2024-02-20T07:57:01.904 DEBUG:teuthology.parallel:result is None 2024-02-20T07:57:01.904 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-20T07:57:01.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-20T07:57:02.165 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:02.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:02 smithi186 ceph-mon[36772]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:02 smithi055 ceph-mon[29254]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:03.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 -- 172.21.15.55:0/3942341733 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 msgr2=0x7f7624103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:03.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 --2- 172.21.15.55:0/3942341733 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624103320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7614009ce0 tx=0x7f7614004d10 comp rx=0 tx=0).stop 2024-02-20T07:57:03.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 -- 172.21.15.55:0/3942341733 shutdown_connections 2024-02-20T07:57:03.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 --2- 172.21.15.55:0/3942341733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76241038f0 0x7f76240fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 --2- 172.21.15.55:0/3942341733 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624103320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 -- 172.21.15.55:0/3942341733 >> 172.21.15.55:0/3942341733 conn(0x7f76240f8ae0 msgr2=0x7f76240faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 -- 172.21.15.55:0/3942341733 shutdown_connections 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.764+0000 7f76290fc700 1 -- 172.21.15.55:0/3942341733 wait complete. 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f76290fc700 1 Processor -- start 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f76290fc700 1 -- start start 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f76290fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f76290fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76241038f0 0x7f7624114130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f76290fc700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7624114780 con 0x7f7624102f50 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f76290fc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762410dad0 con 0x7f76241038f0 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f7622d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f7622d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624113bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57616/0 (socket says 172.21.15.55:57616) 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f7622d9d700 1 -- 172.21.15.55:0/3090186467 learned_addr learned my addr 172.21.15.55:0/3090186467 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f762259c700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76241038f0 0x7f7624114130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.765+0000 7f7622d9d700 1 -- 172.21.15.55:0/3090186467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76241038f0 msgr2=0x7f7624114130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f7622d9d700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76241038f0 0x7f7624114130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f7622d9d700 1 -- 172.21.15.55:0/3090186467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614009940 con 0x7f7624102f50 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f7622d9d700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624113bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7614004b70 tx=0x7f7614004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f761400e040 con 0x7f7624102f50 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762410dd50 con 0x7f7624102f50 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7614004610 con 0x7f7624102f50 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f761401c810 con 0x7f7624102f50 2024-02-20T07:57:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.766+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762410e290 con 0x7f7624102f50 2024-02-20T07:57:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.767+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7624100260 con 0x7f7624102f50 2024-02-20T07:57:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.767+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7614012c00 con 0x7f7624102f50 2024-02-20T07:57:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.768+0000 7f761b7fe700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76100692b0 0x7f761006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.768+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7614088ac0 con 0x7f7624102f50 2024-02-20T07:57:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.768+0000 7f762259c700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76100692b0 0x7f761006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.768+0000 7f762259c700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76100692b0 0x7f761006b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f762410f490 tx=0x7f760c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:03.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.772+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f76140550b0 con 0x7f7624102f50 2024-02-20T07:57:03.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.958+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f762404bed0 con 0x7f76100692b0 2024-02-20T07:57:03.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.959+0000 7f761b7fe700 1 -- 172.21.15.55:0/3090186467 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28183 (secure 0 0 0) 0x7f762404bed0 con 0x7f76100692b0 2024-02-20T07:57:03.961 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76100692b0 msgr2=0x7f761006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76100692b0 0x7f761006b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f762410f490 tx=0x7f760c009040 comp rx=0 tx=0).stop 2024-02-20T07:57:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 msgr2=0x7f7624113bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624113bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7614004b70 tx=0x7f7614004d10 comp rx=0 tx=0).stop 2024-02-20T07:57:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 shutdown_connections 2024-02-20T07:57:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76100692b0 0x7f761006b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7624102f50 0x7f7624113bf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 --2- 172.21.15.55:0/3090186467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76241038f0 0x7f7624114130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.961+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 >> 172.21.15.55:0/3090186467 conn(0x7f76240f8ae0 msgr2=0x7f76240fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.962+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 shutdown_connections 2024-02-20T07:57:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:03.962+0000 7f76290fc700 1 -- 172.21.15.55:0/3090186467 wait complete. 2024-02-20T07:57:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-02-20T07:57:04.226 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:04 smithi055 ceph-mon[29254]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:04.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:04 smithi186 ceph-mon[36772]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:04.578 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-02-20T07:57:03.289472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002769"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-20T07:56:11.170192+0000","last_change":"2024-02-20T07:55:46.255082+0000","last_active":"2024-02-20T07:56:11.170192+0000","last_peered":"2024-02-20T07:56:11.170192+0000","last_clean":"2024-02-20T07:56:11.170192+0000","last_became_active":"2024-02-20T07:55:46.254795+0000","last_became_peered":"2024-02-20T07:55:46.254795+0000","last_unstale":"2024-02-20T07:56:11.170192+0000","last_undegraded":"2024-02-20T07:56:11.170192+0000","last_fullsized":"2024-02-20T07:56:11.170192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-20T07:54:59.573626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-20T07:54:59.573626+0000","last_clean_scrub_stamp":"2024-02-20T07:54:59.573626+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T16:22:39.348638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.434},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.317}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Feb 20 07:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Tue Feb 20 07:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297640,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447512,"statfs":{"total":95995035648,"available":95690252288,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.23300000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.26500000000000001}]},{"osd":3,"last update":"Tue Feb 20 07:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.60399999999999998}]},{"osd":4,"last update":"Tue Feb 20 07:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 20 07:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.317,"15min":0.311},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32100000000000001},"min":{"1min":0.26700000000000002,"5min":0.193,"15min":0.193},"max":{"1min":0.47099999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.28999999999999998}]},{"osd":2,"last update":"Tue Feb 20 07:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.27400000000000002}]},{"osd":3,"last update":"Tue Feb 20 07:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.58799999999999997}]},{"osd":4,"last update":"Tue Feb 20 07:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:56:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.33200000000000002,"15min":0.33100000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55000000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.307}]},{"osd":2,"last update":"Tue Feb 20 07:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.27200000000000002}]},{"osd":3,"last update":"Tue Feb 20 07:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59499999999999997}]},{"osd":4,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-20T07:57:04.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-20T07:57:04.841 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:05.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:05 smithi055 ceph-mon[29254]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:57:05.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:05 smithi186 ceph-mon[36772]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:57:06.200 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:06 smithi055 ceph-mon[29254]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:06.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:06 smithi186 ceph-mon[36772]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 -- 172.21.15.55:0/945033513 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 msgr2=0x7f19580fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 --2- 172.21.15.55:0/945033513 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f19580fbea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f194c004940 tx=0x7f194c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 -- 172.21.15.55:0/945033513 shutdown_connections 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 --2- 172.21.15.55:0/945033513 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f19580fbea0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 --2- 172.21.15.55:0/945033513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1958101e40 0x7f1958102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 -- 172.21.15.55:0/945033513 >> 172.21.15.55:0/945033513 conn(0x7f1958071d90 msgr2=0x7f1958072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 -- 172.21.15.55:0/945033513 shutdown_connections 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 -- 172.21.15.55:0/945033513 wait complete. 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 Processor -- start 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.445+0000 7f196043e700 1 -- start start 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f196043e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1958101e40 0x7f1958112970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f196043e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f1958112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f196043e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1958113500 con 0x7f19581027e0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f196043e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195810c830 con 0x7f1958101e40 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f1958112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f1958112eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57636/0 (socket says 172.21.15.55:57636) 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 -- 172.21.15.55:0/3214771457 learned_addr learned my addr 172.21.15.55:0/3214771457 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195e1da700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1958101e40 0x7f1958112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 -- 172.21.15.55:0/3214771457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1958101e40 msgr2=0x7f1958112970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1958101e40 0x7f1958112970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 -- 172.21.15.55:0/3214771457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c005040 con 0x7f19581027e0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f195d9d9700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f1958112eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f194c000c00 tx=0x7f194c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f194c00e070 con 0x7f19581027e0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195810cab0 con 0x7f19581027e0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f194c013e50 con 0x7f19581027e0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f194c023890 con 0x7f19581027e0 2024-02-20T07:57:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.446+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195810cff0 con 0x7f19581027e0 2024-02-20T07:57:06.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.448+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19580738b0 con 0x7f19581027e0 2024-02-20T07:57:06.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.448+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f194c01a480 con 0x7f19581027e0 2024-02-20T07:57:06.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.448+0000 7f1956ffd700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f19400692b0 0x7f194006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:06.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.448+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f194c08aa10 con 0x7f19581027e0 2024-02-20T07:57:06.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.448+0000 7f195e1da700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f19400692b0 0x7f194006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:06.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.449+0000 7f195e1da700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f19400692b0 0x7f194006b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1948001470 tx=0x7f194800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:06.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.453+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f194c05aa70 con 0x7f19581027e0 2024-02-20T07:57:06.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.639+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f195804bed0 con 0x7f19400692b0 2024-02-20T07:57:06.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.641+0000 7f1956ffd700 1 -- 172.21.15.55:0/3214771457 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28140 (secure 0 0 0) 0x7f195804bed0 con 0x7f19400692b0 2024-02-20T07:57:06.642 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:06.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f19400692b0 msgr2=0x7f194006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:06.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f19400692b0 0x7f194006b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1948001470 tx=0x7f194800b040 comp rx=0 tx=0).stop 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 msgr2=0x7f1958112eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f1958112eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f194c000c00 tx=0x7f194c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 shutdown_connections 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1958101e40 0x7f1958112970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f19400692b0 0x7f194006b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 --2- 172.21.15.55:0/3214771457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19581027e0 0x7f1958112eb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 >> 172.21.15.55:0/3214771457 conn(0x7f1958071d90 msgr2=0x7f19580fad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 shutdown_connections 2024-02-20T07:57:06.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:06.643+0000 7f196043e700 1 -- 172.21.15.55:0/3214771457 wait complete. 2024-02-20T07:57:06.646 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-02-20T07:57:07.260 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:07 smithi055 ceph-mon[29254]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:57:07.286 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-02-20T07:57:05.289868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002653"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-20T07:56:11.170192+0000","last_change":"2024-02-20T07:55:46.255082+0000","last_active":"2024-02-20T07:56:11.170192+0000","last_peered":"2024-02-20T07:56:11.170192+0000","last_clean":"2024-02-20T07:56:11.170192+0000","last_became_active":"2024-02-20T07:55:46.254795+0000","last_became_peered":"2024-02-20T07:55:46.254795+0000","last_unstale":"2024-02-20T07:56:11.170192+0000","last_undegraded":"2024-02-20T07:56:11.170192+0000","last_fullsized":"2024-02-20T07:56:11.170192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-20T07:54:59.573626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-20T07:54:59.573626+0000","last_clean_scrub_stamp":"2024-02-20T07:54:59.573626+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T16:22:39.348638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.434},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.317}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Feb 20 07:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Tue Feb 20 07:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297640,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447512,"statfs":{"total":95995035648,"available":95690252288,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.36699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45900000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.309},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.496,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.26200000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.38,"15min":0.38700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51500000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.40999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.52900000000000003,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.28199999999999997}]},{"osd":3,"last update":"Tue Feb 20 07:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Feb 20 07:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 20 07:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.317,"15min":0.311},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.373},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32100000000000001},"min":{"1min":0.26700000000000002,"5min":0.193,"15min":0.193},"max":{"1min":0.47099999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.38900000000000001}]},{"osd":2,"last update":"Tue Feb 20 07:57:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.38300000000000001,"15min":0.39100000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.40200000000000002,"15min":0.40999999999999998},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.496,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.44800000000000001}]},{"osd":3,"last update":"Tue Feb 20 07:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.57999999999999996}]},{"osd":4,"last update":"Tue Feb 20 07:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.498},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:56:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.33200000000000002,"15min":0.33100000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55000000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.307}]},{"osd":2,"last update":"Tue Feb 20 07:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.27200000000000002}]},{"osd":3,"last update":"Tue Feb 20 07:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59499999999999997}]},{"osd":4,"last update":"Tue Feb 20 07:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-20T07:57:07.287 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-20T07:57:07.287 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-20T07:57:07.287 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-20T07:57:07.287 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph health --format=json 2024-02-20T07:57:07.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:07 smithi186 ceph-mon[36772]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:57:07.548 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:08.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:08 smithi186 ceph-mon[36772]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:08 smithi055 ceph-mon[29254]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.155+0000 7f18f795d700 1 -- 172.21.15.55:0/3533988255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 msgr2=0x7f18f01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.155+0000 7f18f795d700 1 --2- 172.21.15.55:0/3533988255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f01092e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18e0004940 tx=0x7f18e0009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.155+0000 7f18f795d700 1 -- 172.21.15.55:0/3533988255 shutdown_connections 2024-02-20T07:57:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.155+0000 7f18f795d700 1 --2- 172.21.15.55:0/3533988255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f01092e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.155+0000 7f18f795d700 1 --2- 172.21.15.55:0/3533988255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18f00702b0 0x7f18f0070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.155+0000 7f18f795d700 1 -- 172.21.15.55:0/3533988255 >> 172.21.15.55:0/3533988255 conn(0x7f18f00f8ae0 msgr2=0x7f18f00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 -- 172.21.15.55:0/3533988255 shutdown_connections 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 -- 172.21.15.55:0/3533988255 wait complete. 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 Processor -- start 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 -- start start 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18f00702b0 0x7f18f0115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f0116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f01168e0 con 0x7f18f0070bc0 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f795d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f010fbf0 con 0x7f18f00702b0 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f4ef8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f0116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f4ef8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f0116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57650/0 (socket says 172.21.15.55:57650) 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.156+0000 7f18f4ef8700 1 -- 172.21.15.55:0/3310405315 learned_addr learned my addr 172.21.15.55:0/3310405315 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f56f9700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18f00702b0 0x7f18f0115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f4ef8700 1 -- 172.21.15.55:0/3310405315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18f00702b0 msgr2=0x7f18f0115d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f4ef8700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18f00702b0 0x7f18f0115d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f4ef8700 1 -- 172.21.15.55:0/3310405315 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0005040 con 0x7f18f0070bc0 2024-02-20T07:57:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f4ef8700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f0116290 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b3b0 tx=0x7f18e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18e000e040 con 0x7f18f0070bc0 2024-02-20T07:57:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18e000bbf0 con 0x7f18f0070bc0 2024-02-20T07:57:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18e001c3f0 con 0x7f18f0070bc0 2024-02-20T07:57:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f010fe70 con 0x7f18f0070bc0 2024-02-20T07:57:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.157+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f01103b0 con 0x7f18f0070bc0 2024-02-20T07:57:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.158+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f01069a0 con 0x7f18f0070bc0 2024-02-20T07:57:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.159+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f18e000bd60 con 0x7f18f0070bc0 2024-02-20T07:57:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.159+0000 7f18e5ffb700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f18dc0692a0 0x7f18dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.159+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f18e00880a0 con 0x7f18f0070bc0 2024-02-20T07:57:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.159+0000 7f18f56f9700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f18dc0692a0 0x7f18dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:09.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.159+0000 7f18f56f9700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f18dc0692a0 0x7f18dc06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0041d0 tx=0x7f18ec0043c0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:09.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.163+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f18e0058240 con 0x7f18f0070bc0 2024-02-20T07:57:09.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.397+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f18f00fc6e0 con 0x7f18f0070bc0 2024-02-20T07:57:09.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.398+0000 7f18e5ffb700 1 -- 172.21.15.55:0/3310405315 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f18e0057dd0 con 0x7f18f0070bc0 2024-02-20T07:57:09.399 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:09.399 INFO:teuthology.orchestra.run.smithi055.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.399+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f18dc0692a0 msgr2=0x7f18dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.399+0000 7f18f795d700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f18dc0692a0 0x7f18dc06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0041d0 tx=0x7f18ec0043c0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.399+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 msgr2=0x7f18f0116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.399+0000 7f18f795d700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f0116290 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b3b0 tx=0x7f18e0009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 shutdown_connections 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18f00702b0 0x7f18f0115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f18dc0692a0 0x7f18dc06b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 --2- 172.21.15.55:0/3310405315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18f0070bc0 0x7f18f0116290 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 >> 172.21.15.55:0/3310405315 conn(0x7f18f00f8ae0 msgr2=0x7f18f0103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 shutdown_connections 2024-02-20T07:57:09.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:09.400+0000 7f18f795d700 1 -- 172.21.15.55:0/3310405315 wait complete. 2024-02-20T07:57:10.070 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-20T07:57:10.070 INFO:tasks.cephadm:Setup complete, yielding 2024-02-20T07:57:10.071 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:57:10.080 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-20T07:57:10.081 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:10.110 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:10.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-20T07:57:10.373 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:10.621 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:10 smithi055 ceph-mon[29254]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:10.621 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:10 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3310405315' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-20T07:57:10.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:10 smithi186 ceph-mon[36772]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:10.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:10 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3310405315' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 -- 172.21.15.55:0/2578442675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fbe40 msgr2=0x7f4ee00fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 --2- 172.21.15.55:0/2578442675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fc250 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4edc004940 tx=0x7f4edc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 -- 172.21.15.55:0/2578442675 shutdown_connections 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 --2- 172.21.15.55:0/2578442675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 --2- 172.21.15.55:0/2578442675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fc250 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 -- 172.21.15.55:0/2578442675 >> 172.21.15.55:0/2578442675 conn(0x7f4ee00f7520 msgr2=0x7f4ee00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 -- 172.21.15.55:0/2578442675 shutdown_connections 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.015+0000 7f4ee8596700 1 -- 172.21.15.55:0/2578442675 wait complete. 2024-02-20T07:57:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 Processor -- start 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 -- start start 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee00fe980 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee00feaf0 con 0x7f4ee00fbe40 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57664/0 (socket says 172.21.15.55:57664) 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 -- 172.21.15.55:0/3588802554 learned_addr learned my addr 172.21.15.55:0/3588802554 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee6332700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 -- 172.21.15.55:0/3588802554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fbe40 msgr2=0x7f4ee00fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 -- 172.21.15.55:0/3588802554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc005040 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee6332700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee5b31700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fe380 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0002a50 tx=0x7f4ed00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ed0007cc0 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.016+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee00fedd0 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.017+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ed000f040 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.017+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ed000db70 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.017+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee0115980 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.017+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee004bed0 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.018+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4ed000dd10 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.018+0000 7f4ed6ffd700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ecc0692a0 0x7f4ecc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:12.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.018+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ed00865c0 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.018+0000 7f4ee6332700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ecc0692a0 0x7f4ecc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:12.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.019+0000 7f4ee6332700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ecc0692a0 0x7f4ecc06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00b3b0 tx=0x7f4edc009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:12.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.023+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4ed0056640 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.217+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f4ee0104950 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.221+0000 7f4ed6ffd700 1 -- 172.21.15.55:0/3588802554 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f4ed00561d0 con 0x7f4ee00fcfb0 2024-02-20T07:57:12.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ecc0692a0 msgr2=0x7f4ecc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:12.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ecc0692a0 0x7f4ecc06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00b3b0 tx=0x7f4edc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 msgr2=0x7f4ee00fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fe380 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0002a50 tx=0x7f4ed00046b0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 shutdown_connections 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee00fbe40 0x7f4ee00fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ecc0692a0 0x7f4ecc06b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 --2- 172.21.15.55:0/3588802554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee00fcfb0 0x7f4ee00fe380 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 >> 172.21.15.55:0/3588802554 conn(0x7f4ee00f7520 msgr2=0x7f4ee0065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 shutdown_connections 2024-02-20T07:57:12.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:12.223+0000 7f4ee8596700 1 -- 172.21.15.55:0/3588802554 wait complete. 2024-02-20T07:57:12.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:12 smithi055 ceph-mon[29254]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:12.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:12 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3588802554' entity='client.admin' 2024-02-20T07:57:12.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:12 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:12.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:12 smithi186 ceph-mon[36772]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:12.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:12 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3588802554' entity='client.admin' 2024-02-20T07:57:12.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:12 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:12.899 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:57:12.908 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-20T07:57:12.908 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:12.922 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:12.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch status' 2024-02-20T07:57:13.181 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:13.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:13 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:13.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:13 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:13.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:13 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:13.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:13 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:13.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:13 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:13.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:13 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:13.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:13 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:13.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:13 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:14.588 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:14 smithi055 ceph-mon[29254]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:14.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:14 smithi186 ceph-mon[36772]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.847+0000 7f768d60e700 1 -- 172.21.15.55:0/4104346925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 msgr2=0x7f76880fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.847+0000 7f768d60e700 1 --2- 172.21.15.55:0/4104346925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f76880fd320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7678004940 tx=0x7f7678009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.847+0000 7f768d60e700 1 -- 172.21.15.55:0/4104346925 shutdown_connections 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.847+0000 7f768d60e700 1 --2- 172.21.15.55:0/4104346925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880fe0e0 0x7f76880fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.847+0000 7f768d60e700 1 --2- 172.21.15.55:0/4104346925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f76880fd320 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.847+0000 7f768d60e700 1 -- 172.21.15.55:0/4104346925 >> 172.21.15.55:0/4104346925 conn(0x7f76880f8620 msgr2=0x7f76880faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 -- 172.21.15.55:0/4104346925 shutdown_connections 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 -- 172.21.15.55:0/4104346925 wait complete. 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 Processor -- start 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 -- start start 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f768810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:14.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880fe0e0 0x7f76881100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76881106c0 con 0x7f76880fcef0 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.848+0000 7f768d60e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7688110f60 con 0x7f76880fe0e0 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f768810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f768810fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56138/0 (socket says 172.21.15.55:56138) 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 -- 172.21.15.55:0/3430723265 learned_addr learned my addr 172.21.15.55:0/3430723265 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f76867fc700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880fe0e0 0x7f76881100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 -- 172.21.15.55:0/3430723265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880fe0e0 msgr2=0x7f76881100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880fe0e0 0x7f76881100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 -- 172.21.15.55:0/3430723265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678005040 con 0x7f76880fcef0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f7686ffd700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f768810fb80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f767800b3b0 tx=0x7f7678009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f767800e040 con 0x7f76880fcef0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f767800bcc0 con 0x7f76880fcef0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.849+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76881111e0 con 0x7f76880fcef0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.850+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f767801c630 con 0x7f76880fcef0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.850+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7688111720 con 0x7f76880fcef0 2024-02-20T07:57:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.851+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7688102210 con 0x7f76880fcef0 2024-02-20T07:57:14.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.852+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7678013980 con 0x7f76880fcef0 2024-02-20T07:57:14.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.852+0000 7f766f7fe700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76700692a0 0x7f767006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:14.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.852+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7678088200 con 0x7f76880fcef0 2024-02-20T07:57:14.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.852+0000 7f76867fc700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76700692a0 0x7f767006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:14.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.852+0000 7f76867fc700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76700692a0 0x7f767006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f767c0041d0 tx=0x7f767c0043c0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:14.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:14.856+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7678058450 con 0x7f76880fcef0 2024-02-20T07:57:15.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.061+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f768802b3b0 con 0x7f76700692a0 2024-02-20T07:57:15.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.062+0000 7f766f7fe700 1 -- 172.21.15.55:0/3430723265 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f768802b3b0 con 0x7f76700692a0 2024-02-20T07:57:15.063 INFO:teuthology.orchestra.run.smithi055.stdout:Backend: cephadm 2024-02-20T07:57:15.063 INFO:teuthology.orchestra.run.smithi055.stdout:Available: Yes 2024-02-20T07:57:15.063 INFO:teuthology.orchestra.run.smithi055.stdout:Paused: No 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76700692a0 msgr2=0x7f767006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76700692a0 0x7f767006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f767c0041d0 tx=0x7f767c0043c0 comp rx=0 tx=0).stop 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 msgr2=0x7f768810fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f768810fb80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f767800b3b0 tx=0x7f7678009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 shutdown_connections 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f76700692a0 0x7f767006b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76880fcef0 0x7f768810fb80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 --2- 172.21.15.55:0/3430723265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880fe0e0 0x7f76881100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 >> 172.21.15.55:0/3430723265 conn(0x7f76880f8620 msgr2=0x7f76880ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 shutdown_connections 2024-02-20T07:57:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:15.064+0000 7f768d60e700 1 -- 172.21.15.55:0/3430723265 wait complete. 2024-02-20T07:57:15.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-20T07:57:16.038 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:16.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:16 smithi055 ceph-mon[29254]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:16.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:16 smithi055 ceph-mon[29254]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:16.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:16 smithi186 ceph-mon[36772]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:16.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:16 smithi186 ceph-mon[36772]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:17.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.744+0000 7f66eb069700 1 -- 172.21.15.55:0/3881409934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fbe40 msgr2=0x7f66e40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:17.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.744+0000 7f66eb069700 1 --2- 172.21.15.55:0/3881409934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fbe40 0x7f66e40fc250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f66d8004940 tx=0x7f66d8009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:17.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.744+0000 7f66eb069700 1 -- 172.21.15.55:0/3881409934 shutdown_connections 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.744+0000 7f66eb069700 1 --2- 172.21.15.55:0/3881409934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66e40fcfb0 0x7f66e40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.744+0000 7f66eb069700 1 --2- 172.21.15.55:0/3881409934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fbe40 0x7f66e40fc250 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.744+0000 7f66eb069700 1 -- 172.21.15.55:0/3881409934 >> 172.21.15.55:0/3881409934 conn(0x7f66e40f7520 msgr2=0x7f66e40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.745+0000 7f66eb069700 1 -- 172.21.15.55:0/3881409934 shutdown_connections 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.745+0000 7f66eb069700 1 -- 172.21.15.55:0/3881409934 wait complete. 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.745+0000 7f66eb069700 1 Processor -- start 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.745+0000 7f66eb069700 1 -- start start 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.745+0000 7f66eb069700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66e40fbe40 0x7f66e410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.745+0000 7f66eb069700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 0x7f66e410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66eb069700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e410e500 con 0x7f66e40fcfb0 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66eb069700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e410ed40 con 0x7f66e40fbe40 2024-02-20T07:57:17.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 0x7f66e410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 0x7f66e410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56160/0 (socket says 172.21.15.55:56160) 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 -- 172.21.15.55:0/3042968924 learned_addr learned my addr 172.21.15.55:0/3042968924 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 -- 172.21.15.55:0/3042968924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66e40fbe40 msgr2=0x7f66e410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66e40fbe40 0x7f66e410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 -- 172.21.15.55:0/3042968924 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8005040 con 0x7f66e40fcfb0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e3fff700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 0x7f66e410de70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f66d000b790 tx=0x7f66d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66d000f770 con 0x7f66e40fcfb0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66d000fda0 con 0x7f66e40fcfb0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66d000d310 con 0x7f66e40fcfb0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e410f020 con 0x7f66e40fcfb0 2024-02-20T07:57:17.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.746+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e410f560 con 0x7f66e40fcfb0 2024-02-20T07:57:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.747+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e404bed0 con 0x7f66e40fcfb0 2024-02-20T07:57:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.748+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f66d0010420 con 0x7f66e40fcfb0 2024-02-20T07:57:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.748+0000 7f66e17fa700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f66d40692a0 0x7f66d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.748+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f66d0086e70 con 0x7f66e40fcfb0 2024-02-20T07:57:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.748+0000 7f66e8e05700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f66d40692a0 0x7f66d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.749+0000 7f66e8e05700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f66d40692a0 0x7f66d406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f66d8004f90 tx=0x7f66d8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:17.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.752+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f66d0056f70 con 0x7f66e40fcfb0 2024-02-20T07:57:17.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.954+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66e410fa80 con 0x7f66d40692a0 2024-02-20T07:57:17.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.962+0000 7f66e17fa700 1 -- 172.21.15.55:0/3042968924 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f66e410fa80 con 0x7f66d40692a0 2024-02-20T07:57:17.963 INFO:teuthology.orchestra.run.smithi055.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-20T07:57:17.963 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager.smithi055 smithi055 *:9093,9094 running (3m) 2m ago 5m 25.7M - 0.25.0 c8568f914cd2 78ec2fe9d5fe 2024-02-20T07:57:17.963 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi055 smithi055 running (5m) 2m ago 5m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 cb7d541d7455 2024-02-20T07:57:17.963 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi186 smithi186 running (4m) 69s ago 4m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 bd75f3a6c061 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:grafana.smithi055 smithi055 *:3000 running (3m) 2m ago 5m 77.9M - 9.4.7 954c08fa6188 064f316099b8 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi055.stjper smithi055 *:9283 running (6m) 2m ago 6m 469M - 17.2.7-521-gc80e7168 10b0c032c913 eaf5dc05d1a0 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi186.shtgck smithi186 *:8443,9283 running (4m) 69s ago 4m 433M - 17.2.7-521-gc80e7168 10b0c032c913 26e8086a7249 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi055 smithi055 running (6m) 2m ago 6m 44.1M 2048M 17.2.7-521-gc80e7168 10b0c032c913 a759d641dbe8 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi186 smithi186 running (4m) 69s ago 4m 40.6M 2048M 17.2.7-521-gc80e7168 10b0c032c913 6b7e709ad8c3 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi055 smithi055 *:9100 running (5m) 2m ago 5m 15.5M - 1.5.0 0da6a335fe13 fd6cd5796b28 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi186 smithi186 *:9100 running (4m) 69s ago 4m 16.5M - 1.5.0 0da6a335fe13 e7382d1097fe 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 smithi055 running (3m) 2m ago 3m 37.9M 4289M 17.2.7-521-gc80e7168 10b0c032c913 cbfa0b8c0da1 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 smithi055 running (2m) 2m ago 2m 37.8M 4289M 17.2.7-521-gc80e7168 10b0c032c913 eb82fe3e8a01 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 smithi055 running (2m) 2m ago 2m 11.1M 4289M 17.2.7-521-gc80e7168 10b0c032c913 4203cbabc127 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 smithi186 running (2m) 69s ago 2m 38.5M 5312M 17.2.7-521-gc80e7168 10b0c032c913 d24330bb95af 2024-02-20T07:57:17.964 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4 smithi186 running (98s) 69s ago 98s 37.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 a8c7cce19d0f 2024-02-20T07:57:17.965 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5 smithi186 running (74s) 69s ago 73s 10.7M 5312M 17.2.7-521-gc80e7168 10b0c032c913 7583a0fcf7c0 2024-02-20T07:57:17.965 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus.smithi055 smithi055 *:9095 running (3m) 2m ago 4m 38.2M - 2.43.0 a07b618ecd1d d66a8225e95d 2024-02-20T07:57:17.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f66d40692a0 msgr2=0x7f66d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:17.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f66d40692a0 0x7f66d406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f66d8004f90 tx=0x7f66d8009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:17.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 msgr2=0x7f66e410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 0x7f66e410de70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f66d000b790 tx=0x7f66d0009250 comp rx=0 tx=0).stop 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 shutdown_connections 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66e40fbe40 0x7f66e410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f66d40692a0 0x7f66d406b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 --2- 172.21.15.55:0/3042968924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66e40fcfb0 0x7f66e410de70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 >> 172.21.15.55:0/3042968924 conn(0x7f66e40f7520 msgr2=0x7f66e4065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 shutdown_connections 2024-02-20T07:57:17.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:17.964+0000 7f66eb069700 1 -- 172.21.15.55:0/3042968924 wait complete. 2024-02-20T07:57:18.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-20T07:57:18.638 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:18 smithi055 ceph-mon[29254]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:18.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:18 smithi186 ceph-mon[36772]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:18.870 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:19.675 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:19 smithi055 ceph-mon[29254]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:19.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:19 smithi186 ceph-mon[36772]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:20.479 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:20 smithi055 ceph-mon[29254]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 -- 172.21.15.55:0/1205351886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fcef0 msgr2=0x7f47400fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 --2- 172.21.15.55:0/1205351886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fcef0 0x7f47400fd320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4738009ce0 tx=0x7f4738004d10 comp rx=0 tx=0).stop 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 -- 172.21.15.55:0/1205351886 shutdown_connections 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 --2- 172.21.15.55:0/1205351886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fe0e0 0x7f47400fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 --2- 172.21.15.55:0/1205351886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fcef0 0x7f47400fd320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 -- 172.21.15.55:0/1205351886 >> 172.21.15.55:0/1205351886 conn(0x7f47400f8620 msgr2=0x7f47400faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:20.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 -- 172.21.15.55:0/1205351886 shutdown_connections 2024-02-20T07:57:20.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 -- 172.21.15.55:0/1205351886 wait complete. 2024-02-20T07:57:20.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 Processor -- start 2024-02-20T07:57:20.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.490+0000 7f4745899700 1 -- start start 2024-02-20T07:57:20.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f4745899700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fcef0 0x7f4740071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f4745899700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 0x7f47400702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f4745899700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47400708e0 con 0x7f47400fe0e0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f4745899700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4740070a50 con 0x7f47400fcef0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 0x7f47400702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 0x7f47400702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56178/0 (socket says 172.21.15.55:56178) 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 -- 172.21.15.55:0/4010989259 learned_addr learned my addr 172.21.15.55:0/4010989259 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473effd700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fcef0 0x7f4740071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 -- 172.21.15.55:0/4010989259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fcef0 msgr2=0x7f4740071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fcef0 0x7f4740071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 -- 172.21.15.55:0/4010989259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4738009940 con 0x7f47400fe0e0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473effd700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fcef0 0x7f4740071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f473e7fc700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 0x7f47400702e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f472800c8c0 tx=0x7f47280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f472800a980 con 0x7f47400fe0e0 2024-02-20T07:57:20.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f472800ce60 con 0x7f47400fe0e0 2024-02-20T07:57:20.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4740070d30 con 0x7f47400fe0e0 2024-02-20T07:57:20.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.491+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47280115e0 con 0x7f47400fe0e0 2024-02-20T07:57:20.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.492+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4740115b10 con 0x7f47400fe0e0 2024-02-20T07:57:20.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.492+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474004bed0 con 0x7f47400fe0e0 2024-02-20T07:57:20.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.493+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4728012400 con 0x7f47400fe0e0 2024-02-20T07:57:20.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.493+0000 7f47377fe700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f472c0692a0 0x7f472c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:20.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.494+0000 7f473effd700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f472c0692a0 0x7f472c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:20.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.494+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4728086f50 con 0x7f47400fe0e0 2024-02-20T07:57:20.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.494+0000 7f473effd700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f472c0692a0 0x7f472c06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4738000f80 tx=0x7f4738004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:20.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.498+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4728057050 con 0x7f47400fe0e0 2024-02-20T07:57:20.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.694+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4740102ac0 con 0x7f472c0692a0 2024-02-20T07:57:20.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.699+0000 7f47377fe700 1 -- 172.21.15.55:0/4010989259 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f4740102ac0 con 0x7f472c0692a0 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:crash 2/2 2m ago 6m * 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:mgr 2/2 2m ago 6m count:2 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:mon 2/2 2m ago 5m smithi055:172.21.15.55=smithi055;smithi186:172.21.15.186=smithi186;count:2 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:osd 6 2m ago - 2024-02-20T07:57:20.700 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f472c0692a0 msgr2=0x7f472c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f472c0692a0 0x7f472c06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4738000f80 tx=0x7f4738004d10 comp rx=0 tx=0).stop 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 msgr2=0x7f47400702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 0x7f47400702e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f472800c8c0 tx=0x7f47280044c0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 shutdown_connections 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47400fcef0 0x7f4740071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f472c0692a0 0x7f472c06b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 --2- 172.21.15.55:0/4010989259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47400fe0e0 0x7f47400702e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 >> 172.21.15.55:0/4010989259 conn(0x7f47400f8620 msgr2=0x7f47401013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 shutdown_connections 2024-02-20T07:57:20.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:20.701+0000 7f4745899700 1 -- 172.21.15.55:0/4010989259 wait complete. 2024-02-20T07:57:20.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:20 smithi186 ceph-mon[36772]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:21.408 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-20T07:57:21.716 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:22.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:22 smithi186 ceph-mon[36772]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:22.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:22 smithi186 ceph-mon[36772]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:22.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:22 smithi055 ceph-mon[29254]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:22.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:22 smithi055 ceph-mon[29254]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 -- 172.21.15.55:0/742443863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 msgr2=0x7f8b44070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 --2- 172.21.15.55:0/742443863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b44070cb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c004940 tx=0x7f8b2c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 -- 172.21.15.55:0/742443863 shutdown_connections 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 --2- 172.21.15.55:0/742443863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b44070cb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 --2- 172.21.15.55:0/742443863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b44071e90 0x7f8b44070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 -- 172.21.15.55:0/742443863 >> 172.21.15.55:0/742443863 conn(0x7f8b440f8620 msgr2=0x7f8b440faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 -- 172.21.15.55:0/742443863 shutdown_connections 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.290+0000 7f8b49322700 1 -- 172.21.15.55:0/742443863 wait complete. 2024-02-20T07:57:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b49322700 1 Processor -- start 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b49322700 1 -- start start 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b49322700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b4410fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b49322700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b44071e90 0x7f8b44110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b49322700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44110680 con 0x7f8b44070840 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b49322700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44110f20 con 0x7f8b44071e90 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b4410fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b4410fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45128/0 (socket says 172.21.15.55:45128) 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 -- 172.21.15.55:0/3774818597 learned_addr learned my addr 172.21.15.55:0/3774818597 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b427fc700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b44071e90 0x7f8b44110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 -- 172.21.15.55:0/3774818597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b44071e90 msgr2=0x7f8b44110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b44071e90 0x7f8b44110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 -- 172.21.15.55:0/3774818597 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c005040 con 0x7f8b44070840 2024-02-20T07:57:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.291+0000 7f8b42ffd700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b4410fb40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c001d80 tx=0x7f8b3c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.292+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3c007d10 con 0x7f8b44070840 2024-02-20T07:57:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.292+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b3c001e80 con 0x7f8b44070840 2024-02-20T07:57:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.292+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3c014640 con 0x7f8b44070840 2024-02-20T07:57:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.292+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b44111200 con 0x7f8b44070840 2024-02-20T07:57:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.292+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b44111740 con 0x7f8b44070840 2024-02-20T07:57:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.293+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4404bed0 con 0x7f8b44070840 2024-02-20T07:57:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.294+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8b3c00e960 con 0x7f8b44070840 2024-02-20T07:57:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.294+0000 7f8b3b7fe700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f8b300692a0 0x7f8b3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.294+0000 7f8b427fc700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f8b300692a0 0x7f8b3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.294+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8b3c086fc0 con 0x7f8b44070840 2024-02-20T07:57:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.294+0000 7f8b427fc700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f8b300692a0 0x7f8b3006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00b3b0 tx=0x7f8b2c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.298+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8b3c057210 con 0x7f8b44070840 2024-02-20T07:57:23.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.496+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b44104570 con 0x7f8b300692a0 2024-02-20T07:57:23.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.497+0000 7f8b3b7fe700 1 -- 172.21.15.55:0/3774818597 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f8b44104570 con 0x7f8b300692a0 2024-02-20T07:57:23.498 INFO:teuthology.orchestra.run.smithi055.stdout:HOST ADDR LABELS STATUS 2024-02-20T07:57:23.499 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055 172.21.15.55 2024-02-20T07:57:23.499 INFO:teuthology.orchestra.run.smithi055.stdout:smithi186 172.21.15.186 2024-02-20T07:57:23.499 INFO:teuthology.orchestra.run.smithi055.stdout:2 hosts in cluster 2024-02-20T07:57:23.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f8b300692a0 msgr2=0x7f8b3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:23.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f8b300692a0 0x7f8b3006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00b3b0 tx=0x7f8b2c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:23.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 msgr2=0x7f8b4410fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b4410fb40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c001d80 tx=0x7f8b3c009ec0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 shutdown_connections 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f8b300692a0 0x7f8b3006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b44070840 0x7f8b4410fb40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 --2- 172.21.15.55:0/3774818597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b44071e90 0x7f8b44110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 >> 172.21.15.55:0/3774818597 conn(0x7f8b440f8620 msgr2=0x7f8b44102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 shutdown_connections 2024-02-20T07:57:23.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:23.499+0000 7f8b49322700 1 -- 172.21.15.55:0/3774818597 wait complete. 2024-02-20T07:57:24.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-20T07:57:24.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:24 smithi055 ceph-mon[29254]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:24.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:24 smithi055 ceph-mon[29254]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:24.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:24 smithi186 ceph-mon[36772]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:24.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:24 smithi186 ceph-mon[36772]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:24.338 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:26.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 -- 172.21.15.55:0/2535896544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 msgr2=0x7efee80fe650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:26.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 --2- 172.21.15.55:0/2535896544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee80fe650 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efedc004940 tx=0x7efedc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:26.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 -- 172.21.15.55:0/2535896544 shutdown_connections 2024-02-20T07:57:26.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 --2- 172.21.15.55:0/2535896544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee80fe650 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 --2- 172.21.15.55:0/2535896544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee80fcf30 0x7efee80fd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 -- 172.21.15.55:0/2535896544 >> 172.21.15.55:0/2535896544 conn(0x7efee80f86b0 msgr2=0x7efee80faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 -- 172.21.15.55:0/2535896544 shutdown_connections 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 -- 172.21.15.55:0/2535896544 wait complete. 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 Processor -- start 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.032+0000 7efef0381700 1 -- start start 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efef0381700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee80fcf30 0x7efee810d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efef0381700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee810de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efef0381700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee810e440 con 0x7efee80fe1e0 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efef0381700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee810ece0 con 0x7efee80fcf30 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee810de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee810de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45146/0 (socket says 172.21.15.55:45146) 2024-02-20T07:57:26.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 -- 172.21.15.55:0/1821734909 learned_addr learned my addr 172.21.15.55:0/1821734909 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeee11d700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee80fcf30 0x7efee810d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 -- 172.21.15.55:0/1821734909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee80fcf30 msgr2=0x7efee810d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee80fcf30 0x7efee810d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 -- 172.21.15.55:0/1821734909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc005040 con 0x7efee80fe1e0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efeed91c700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee810de40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efedc0093b0 tx=0x7efedc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efedc00e040 con 0x7efee80fe1e0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee810ef60 con 0x7efee80fe1e0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efedc00bcc0 con 0x7efee80fe1e0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.033+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee810f4a0 con 0x7efee80fe1e0 2024-02-20T07:57:26.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.034+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efedc01c630 con 0x7efee80fe1e0 2024-02-20T07:57:26.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.034+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee804bed0 con 0x7efee80fe1e0 2024-02-20T07:57:26.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.035+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7efedc013980 con 0x7efee80fe1e0 2024-02-20T07:57:26.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.035+0000 7efee6ffd700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7efed00692a0 0x7efed006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:26.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.035+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efedc0881f0 con 0x7efee80fe1e0 2024-02-20T07:57:26.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.035+0000 7efeee11d700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7efed00692a0 0x7efed006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:26.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.036+0000 7efeee11d700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7efed00692a0 0x7efed006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efed8007a40 tx=0x7efed8009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:26.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.040+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7efedc0582f0 con 0x7efee80fe1e0 2024-02-20T07:57:26.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.240+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7efee810f8f0 con 0x7efed00692a0 2024-02-20T07:57:26.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.241+0000 7efee6ffd700 1 -- 172.21.15.55:0/1821734909 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7efee810f8f0 con 0x7efed00692a0 2024-02-20T07:57:26.243 INFO:teuthology.orchestra.run.smithi055.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-20T07:57:26.244 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102VS4P0KGN 3726G No 2m ago Has a FileSystem, LVM detected 2024-02-20T07:57:26.244 INFO:teuthology.orchestra.run.smithi055.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 74s ago Has a FileSystem, LVM detected 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.243+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7efed00692a0 msgr2=0x7efed006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.243+0000 7efef0381700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7efed00692a0 0x7efed006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efed8007a40 tx=0x7efed8009da0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.243+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 msgr2=0x7efee810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.243+0000 7efef0381700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee810de40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efedc0093b0 tx=0x7efedc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.243+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 shutdown_connections 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.243+0000 7efef0381700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee80fcf30 0x7efee810d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.244+0000 7efef0381700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7efed00692a0 0x7efed006b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.244+0000 7efef0381700 1 --2- 172.21.15.55:0/1821734909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efee80fe1e0 0x7efee810de40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.244+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 >> 172.21.15.55:0/1821734909 conn(0x7efee80f86b0 msgr2=0x7efee8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.244+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 shutdown_connections 2024-02-20T07:57:26.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:26.244+0000 7efef0381700 1 -- 172.21.15.55:0/1821734909 wait complete. 2024-02-20T07:57:26.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:26 smithi055 ceph-mon[29254]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:26.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:26 smithi186 ceph-mon[36772]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:26.867 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-20T07:57:26.878 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:26.930 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:26.930 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-20T07:57:27.196 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:27.628 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:27 smithi055 ceph-mon[29254]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:27.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:27 smithi186 ceph-mon[36772]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:28.580 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:28 smithi055 ceph-mon[29254]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:28.580 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 -- 172.21.15.55:0/624826179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 msgr2=0x7f5c9c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 --2- 172.21.15.55:0/624826179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c0fd320 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c004940 tx=0x7f5c8c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 -- 172.21.15.55:0/624826179 shutdown_connections 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 --2- 172.21.15.55:0/624826179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 0x7f5c9c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 --2- 172.21.15.55:0/624826179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c0fd320 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 -- 172.21.15.55:0/624826179 >> 172.21.15.55:0/624826179 conn(0x7f5c9c0f8620 msgr2=0x7f5c9c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 -- 172.21.15.55:0/624826179 shutdown_connections 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.799+0000 7f5ca2288700 1 -- 172.21.15.55:0/624826179 wait complete. 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5ca2288700 1 Processor -- start 2024-02-20T07:57:28.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5ca2288700 1 -- start start 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5ca2288700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c10d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5ca2288700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 0x7f5c9c10ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5ca2288700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c10e560 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5ca2288700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c10eda0 con 0x7f5c9c0fe0e0 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5c9bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c10d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5c9bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c10d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45168/0 (socket says 172.21.15.55:45168) 2024-02-20T07:57:28.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.800+0000 7f5c9bfff700 1 -- 172.21.15.55:0/111729533 learned_addr learned my addr 172.21.15.55:0/111729533 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c9b7fe700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 0x7f5c9c10ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c9bfff700 1 -- 172.21.15.55:0/111729533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 msgr2=0x7f5c9c10ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c9bfff700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 0x7f5c9c10ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c9bfff700 1 -- 172.21.15.55:0/111729533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c005040 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c9b7fe700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 0x7f5c9c10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c9bfff700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c10d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00b3b0 tx=0x7f5c8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c00e040 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c8c00bcc0 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c10f020 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c01c630 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.801+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c10f560 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.802+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9c04bed0 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.803+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5c8c0139f0 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.803+0000 7f5c98ff9700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5c840692a0 0x7f5c8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:28.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.803+0000 7f5c9b7fe700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5c840692a0 0x7f5c8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:28.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.803+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5c8c088300 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.804+0000 7f5c9b7fe700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5c840692a0 0x7f5c8406b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5c90006fd0 tx=0x7f5c90008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:28.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:28.808+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5c8c058400 con 0x7f5c9c0fcef0 2024-02-20T07:57:28.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:28 smithi186 ceph-mon[36772]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:28.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:29.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:29.031+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c9c102ac0 con 0x7f5c840692a0 2024-02-20T07:57:29.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:29 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-20T07:57:29.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:29 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-20T07:57:30.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.415+0000 7f5c98ff9700 1 -- 172.21.15.55:0/111729533 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5c9c102ac0 con 0x7f5c840692a0 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.418+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5c840692a0 msgr2=0x7f5c8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5c840692a0 0x7f5c8406b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5c90006fd0 tx=0x7f5c90008040 comp rx=0 tx=0).stop 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 msgr2=0x7f5c9c10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c10d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00b3b0 tx=0x7f5c8c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 shutdown_connections 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5c840692a0 0x7f5c8406b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c9c0fcef0 0x7f5c9c10d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 --2- 172.21.15.55:0/111729533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c9c0fe0e0 0x7f5c9c10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 >> 172.21.15.55:0/111729533 conn(0x7f5c9c0f8620 msgr2=0x7f5c9c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:30.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 shutdown_connections 2024-02-20T07:57:30.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:30.419+0000 7f5ca2288700 1 -- 172.21.15.55:0/111729533 wait complete. 2024-02-20T07:57:30.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:30 smithi055 ceph-mon[29254]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:30.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:30 smithi055 ceph-mon[29254]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:30.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:30 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-20T07:57:30.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:30 smithi055 ceph-mon[29254]: osdmap e41: 6 total, 6 up, 6 in 2024-02-20T07:57:30.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:30 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:57:30.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T07:57:30.395+0000 7fc097539700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-20T07:57:30.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:30 smithi186 ceph-mon[36772]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:57:30.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:30 smithi186 ceph-mon[36772]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:30 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-20T07:57:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:30 smithi186 ceph-mon[36772]: osdmap e41: 6 total, 6 up, 6 in 2024-02-20T07:57:30.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:30 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:57:31.022 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: osdmap e42: 6 total, 6 up, 6 in 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: osdmap e43: 6 total, 6 up, 6 in 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-20T07:57:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: fsmap cephfs:0 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: Saving service mds.cephfs spec with placement count:4 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.ptjwai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.ptjwai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:31.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:31 smithi055 ceph-mon[29254]: Deploying daemon mds.cephfs.smithi055.ptjwai on smithi055 2024-02-20T07:57:31.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-20T07:57:31.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: osdmap e42: 6 total, 6 up, 6 in 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: osdmap e43: 6 total, 6 up, 6 in 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: fsmap cephfs:0 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: Saving service mds.cephfs spec with placement count:4 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.ptjwai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:31.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.ptjwai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:31.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:31.826 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:31 smithi186 ceph-mon[36772]: Deploying daemon mds.cephfs.smithi055.ptjwai on smithi055 2024-02-20T07:57:32.233 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:32.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:32 smithi186 ceph-mon[36772]: pgmap v175: 65 pgs: 6 creating+activating, 8 creating+peering, 21 active+clean, 30 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:32.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:32 smithi186 ceph-mon[36772]: osdmap e44: 6 total, 6 up, 6 in 2024-02-20T07:57:32.693 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:32 smithi055 ceph-mon[29254]: pgmap v175: 65 pgs: 6 creating+activating, 8 creating+peering, 21 active+clean, 30 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:32.693 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:32 smithi055 ceph-mon[29254]: osdmap e44: 6 total, 6 up, 6 in 2024-02-20T07:57:33.717 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:33 smithi055 ceph-mon[29254]: osdmap e45: 6 total, 6 up, 6 in 2024-02-20T07:57:33.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:33 smithi186 ceph-mon[36772]: osdmap e45: 6 total, 6 up, 6 in 2024-02-20T07:57:33.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 -- 172.21.15.55:0/1841100113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb00708c0 msgr2=0x7f6fb0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:33.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/1841100113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb00708c0 0x7f6fb0070d10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c004940 tx=0x7f6f9c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:33.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 -- 172.21.15.55:0/1841100113 shutdown_connections 2024-02-20T07:57:33.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/1841100113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb00708c0 0x7f6fb0070d10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:33.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/1841100113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb0071f10 0x7f6fb0070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 -- 172.21.15.55:0/1841100113 >> 172.21.15.55:0/1841100113 conn(0x7f6fb00f8620 msgr2=0x7f6fb00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 -- 172.21.15.55:0/1841100113 shutdown_connections 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 -- 172.21.15.55:0/1841100113 wait complete. 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 Processor -- start 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.966+0000 7f6fb6a87700 1 -- start start 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6fb6a87700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb00708c0 0x7f6fb0113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6fb6a87700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 0x7f6fb01144d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6fb6a87700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0114b60 con 0x7f6fb0071f10 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6fb6a87700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb01153a0 con 0x7f6fb00708c0 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 0x7f6fb01144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 0x7f6fb01144d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59646/0 (socket says 172.21.15.55:59646) 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 -- 172.21.15.55:0/3886647161 learned_addr learned my addr 172.21.15.55:0/3886647161 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6fb4823700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb00708c0 0x7f6fb0113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 -- 172.21.15.55:0/3886647161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb00708c0 msgr2=0x7f6fb0113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb00708c0 0x7f6fb0113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 -- 172.21.15.55:0/3886647161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c005040 con 0x7f6fb0071f10 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.967+0000 7f6faffff700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 0x7f6fb01144d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0093b0 tx=0x7f6f9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.968+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c00e040 con 0x7f6fb0071f10 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.968+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f9c00bcc0 con 0x7f6fb0071f10 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.968+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb0115620 con 0x7f6fb0071f10 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.968+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c01c630 con 0x7f6fb0071f10 2024-02-20T07:57:33.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.968+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb0115b60 con 0x7f6fb0071f10 2024-02-20T07:57:33.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.969+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb004bed0 con 0x7f6fb0071f10 2024-02-20T07:57:33.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.969+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6f9c013420 con 0x7f6fb0071f10 2024-02-20T07:57:33.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.969+0000 7f6fad7fa700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6fa00692a0 0x7f6fa006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.970+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6f9c0885b0 con 0x7f6fb0071f10 2024-02-20T07:57:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.970+0000 7f6fb4823700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6fa00692a0 0x7f6fa006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.970+0000 7f6fb4823700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6fa00692a0 0x7f6fa006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4006fd0 tx=0x7f6fa4008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:33.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:33.974+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6f9c058370 con 0x7f6fb0071f10 2024-02-20T07:57:34.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.214+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6fb0115fd0 con 0x7f6fb0071f10 2024-02-20T07:57:34.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.214+0000 7f6fad7fa700 1 -- 172.21.15.55:0/3886647161 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f6f9c057f00 con 0x7f6fb0071f10 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:e2 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:legacy client fscid: 1 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:Filesystem 'cephfs' (1) 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:fs_name cephfs 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 2 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-20T07:57:34.216 INFO:teuthology.orchestra.run.smithi055.stdout:created 2024-02-20T07:57:30.395820+0000 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:modified 2024-02-20T07:57:30.395885+0000 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:tableserver 0 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:root 0 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:session_timeout 60 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:session_autoclose 300 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:max_file_size 1099511627776 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:required_client_features {} 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure 0 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure_osd_epoch 0 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:57:34.217 INFO:teuthology.orchestra.run.smithi055.stdout:max_mds 1 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:in 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:up {} 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:failed 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:damaged 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:stopped 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:data_pools [3] 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:metadata_pool 2 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:inline_data disabled 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:balancer 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout:standby_count_wanted 0 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:34.218 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6fa00692a0 msgr2=0x7f6fa006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6fa00692a0 0x7f6fa006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4006fd0 tx=0x7f6fa4008040 comp rx=0 tx=0).stop 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 msgr2=0x7f6fb01144d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 0x7f6fb01144d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0093b0 tx=0x7f6f9c009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 shutdown_connections 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb00708c0 0x7f6fb0113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f6fa00692a0 0x7f6fa006b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 --2- 172.21.15.55:0/3886647161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fb0071f10 0x7f6fb01144d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 >> 172.21.15.55:0/3886647161 conn(0x7f6fb00f8620 msgr2=0x7f6fb00fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:34.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 shutdown_connections 2024-02-20T07:57:34.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:34.217+0000 7f6fb6a87700 1 -- 172.21.15.55:0/3886647161 wait complete. 2024-02-20T07:57:34.221 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 2 2024-02-20T07:57:34.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:34 smithi055 ceph-mon[29254]: pgmap v178: 65 pgs: 6 creating+activating, 8 creating+peering, 36 active+clean, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:34.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:34 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3886647161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:57:34.725 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:34 smithi186 ceph-mon[36772]: pgmap v178: 65 pgs: 6 creating+activating, 8 creating+peering, 36 active+clean, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:34.725 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:34 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3886647161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:57:35.087 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:35.097 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:35.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-02-20T07:57:35.361 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:35.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.xntwaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.xntwaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: Deploying daemon mds.cephfs.smithi186.xntwaj on smithi186 2024-02-20T07:57:35.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6826/3290843304,v1:172.21.15.55:6827/3290843304] up:boot 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: daemon mds.cephfs.smithi055.ptjwai assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: Cluster is now healthy 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: fsmap cephfs:0 1 up:standby 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.ptjwai"}]: dispatch 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: fsmap cephfs:1 {0=cephfs.smithi055.ptjwai=up:creating} 2024-02-20T07:57:35.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:35 smithi186 ceph-mon[36772]: daemon mds.cephfs.smithi055.ptjwai is now active in filesystem cephfs as rank 0 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.xntwaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.xntwaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: Deploying daemon mds.cephfs.smithi186.xntwaj on smithi186 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6826/3290843304,v1:172.21.15.55:6827/3290843304] up:boot 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: daemon mds.cephfs.smithi055.ptjwai assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-20T07:57:35.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:57:35.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: Cluster is now healthy 2024-02-20T07:57:35.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: fsmap cephfs:0 1 up:standby 2024-02-20T07:57:35.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.ptjwai"}]: dispatch 2024-02-20T07:57:35.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: fsmap cephfs:1 {0=cephfs.smithi055.ptjwai=up:creating} 2024-02-20T07:57:35.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:35 smithi055 ceph-mon[29254]: daemon mds.cephfs.smithi055.ptjwai is now active in filesystem cephfs as rank 0 2024-02-20T07:57:36.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:36 smithi186 ceph-mon[36772]: pgmap v179: 65 pgs: 6 creating+activating, 5 creating+peering, 54 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:36.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:36 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6826/3290843304,v1:172.21.15.55:6827/3290843304] up:active 2024-02-20T07:57:36.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:36 smithi186 ceph-mon[36772]: fsmap cephfs:1 {0=cephfs.smithi055.ptjwai=up:active} 2024-02-20T07:57:36.692 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:36 smithi055 ceph-mon[29254]: pgmap v179: 65 pgs: 6 creating+activating, 5 creating+peering, 54 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:57:36.692 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:36 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6826/3290843304,v1:172.21.15.55:6827/3290843304] up:active 2024-02-20T07:57:36.692 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:36 smithi055 ceph-mon[29254]: fsmap cephfs:1 {0=cephfs.smithi055.ptjwai=up:active} 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 -- 172.21.15.55:0/2294446597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 msgr2=0x7f04c80fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 --2- 172.21.15.55:0/2294446597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c80fe700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f04bc004940 tx=0x7f04bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 -- 172.21.15.55:0/2294446597 shutdown_connections 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 --2- 172.21.15.55:0/2294446597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c80fe700 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 --2- 172.21.15.55:0/2294446597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 0x7f04c80fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 -- 172.21.15.55:0/2294446597 >> 172.21.15.55:0/2294446597 conn(0x7f04c80f6ca0 msgr2=0x7f04c80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 -- 172.21.15.55:0/2294446597 shutdown_connections 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 -- 172.21.15.55:0/2294446597 wait complete. 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.904+0000 7f04ce741700 1 Processor -- start 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04ce741700 1 -- start start 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04ce741700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 0x7f04c810d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:36.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04ce741700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c810dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:36.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04ce741700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c810e4c0 con 0x7f04c80fe2b0 2024-02-20T07:57:36.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04ce741700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c810ed60 con 0x7f04c80fb220 2024-02-20T07:57:36.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c810dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:36.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c810dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59688/0 (socket says 172.21.15.55:59688) 2024-02-20T07:57:36.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 -- 172.21.15.55:0/4024006431 learned_addr learned my addr 172.21.15.55:0/4024006431 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:36.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c7fff700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 0x7f04c810d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 -- 172.21.15.55:0/4024006431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 msgr2=0x7f04c810d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 0x7f04c810d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 -- 172.21.15.55:0/4024006431 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc005040 con 0x7f04c80fe2b0 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c7fff700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 0x7f04c810d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c77fe700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c810dec0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009b30 tx=0x7f04bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.905+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04bc021070 con 0x7f04c80fe2b0 2024-02-20T07:57:36.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.906+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c810efe0 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.906+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04bc01c400 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.906+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c810f4c0 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.906+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04bc00e070 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.906+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c804bed0 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.907+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f04bc01ca00 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.907+0000 7f04c4ff9700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f04b00692f0 0x7f04b006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.908+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f04bc0883d0 con 0x7f04c80fe2b0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.908+0000 7f04c7fff700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f04b00692f0 0x7f04b006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:36.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.908+0000 7f04c7fff700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f04b00692f0 0x7f04b006b7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009b80 tx=0x7f04b800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:36.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:36.912+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f04bc0580f0 con 0x7f04c80fe2b0 2024-02-20T07:57:37.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.172+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f04c810f8f0 con 0x7f04c80fe2b0 2024-02-20T07:57:37.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.490+0000 7f04c4ff9700 1 -- 172.21.15.55:0/4024006431 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f04bc004490 con 0x7f04c80fe2b0 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f04b00692f0 msgr2=0x7f04b006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f04b00692f0 0x7f04b006b7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009b80 tx=0x7f04b800b040 comp rx=0 tx=0).stop 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 msgr2=0x7f04c810dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c810dec0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009b30 tx=0x7f04bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 shutdown_connections 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04c80fb220 0x7f04c810d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f04b00692f0 0x7f04b006b7a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 --2- 172.21.15.55:0/4024006431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04c80fe2b0 0x7f04c810dec0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:37.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 >> 172.21.15.55:0/4024006431 conn(0x7f04c80f6ca0 msgr2=0x7f04c81014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:37.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 shutdown_connections 2024-02-20T07:57:37.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:37.493+0000 7f04ce741700 1 -- 172.21.15.55:0/4024006431 wait complete. 2024-02-20T07:57:38.054 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:38.065 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:38.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.sinapd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.sinapd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: Deploying daemon mds.cephfs.smithi055.sinapd on smithi055 2024-02-20T07:57:38.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/4024006431' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 445 B/s wr, 1 op/s 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] up:boot 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/4024006431' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: daemon mds.cephfs.smithi186.xntwaj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: fsmap cephfs:1 {0=cephfs.smithi055.ptjwai=up:active} 1 up:standby 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi186.xntwaj"}]: dispatch 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:creating} 2024-02-20T07:57:38.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:37 smithi055 ceph-mon[29254]: daemon mds.cephfs.smithi186.xntwaj is now active in filesystem cephfs as rank 1 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.sinapd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.sinapd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: Deploying daemon mds.cephfs.smithi055.sinapd on smithi055 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/4024006431' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 445 B/s wr, 1 op/s 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:57:38.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] up:boot 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/4024006431' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: daemon mds.cephfs.smithi186.xntwaj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: fsmap cephfs:1 {0=cephfs.smithi055.ptjwai=up:active} 1 up:standby 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi186.xntwaj"}]: dispatch 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:creating} 2024-02-20T07:57:38.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:37 smithi186 ceph-mon[36772]: daemon mds.cephfs.smithi186.xntwaj is now active in filesystem cephfs as rank 1 2024-02-20T07:57:38.513 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:39.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:39 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] up:active 2024-02-20T07:57:39.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:39 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 2024-02-20T07:57:39.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:39 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] up:active 2024-02-20T07:57:39.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:39 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 -- 172.21.15.55:0/3962309325 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fcef0 msgr2=0x7f96640fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 --2- 172.21.15.55:0/3962309325 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fcef0 0x7f96640fd320 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9654004940 tx=0x7f9654009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 -- 172.21.15.55:0/3962309325 shutdown_connections 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 --2- 172.21.15.55:0/3962309325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96640fe0e0 0x7f96640fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 --2- 172.21.15.55:0/3962309325 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fcef0 0x7f96640fd320 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 -- 172.21.15.55:0/3962309325 >> 172.21.15.55:0/3962309325 conn(0x7f96640f8620 msgr2=0x7f96640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 -- 172.21.15.55:0/3962309325 shutdown_connections 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.500+0000 7f966b543700 1 -- 172.21.15.55:0/3962309325 wait complete. 2024-02-20T07:57:40.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f966b543700 1 Processor -- start 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f966b543700 1 -- start start 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f966b543700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96640fcef0 0x7f966410d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f966b543700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 0x7f966410deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f966b543700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966410e4b0 con 0x7f96640fe0e0 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f966b543700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966410ed50 con 0x7f96640fcef0 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f9668ade700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 0x7f966410deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f9668ade700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 0x7f966410deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59704/0 (socket says 172.21.15.55:59704) 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f9668ade700 1 -- 172.21.15.55:0/1806787694 learned_addr learned my addr 172.21.15.55:0/1806787694 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.501+0000 7f96692df700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96640fcef0 0x7f966410d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.502+0000 7f9668ade700 1 -- 172.21.15.55:0/1806787694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96640fcef0 msgr2=0x7f966410d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.502+0000 7f9668ade700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96640fcef0 0x7f966410d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.502+0000 7f9668ade700 1 -- 172.21.15.55:0/1806787694 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9654005040 con 0x7f96640fe0e0 2024-02-20T07:57:40.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.502+0000 7f9668ade700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 0x7f966410deb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9660002a50 tx=0x7f96600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:40.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.504+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9660007cc0 con 0x7f96640fe0e0 2024-02-20T07:57:40.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.504+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f966000f040 con 0x7f96640fe0e0 2024-02-20T07:57:40.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.504+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966410f030 con 0x7f96640fe0e0 2024-02-20T07:57:40.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.504+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f966000da30 con 0x7f96640fe0e0 2024-02-20T07:57:40.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.505+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966410f570 con 0x7f96640fe0e0 2024-02-20T07:57:40.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.506+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966404bed0 con 0x7f96640fe0e0 2024-02-20T07:57:40.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.506+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f966000db90 con 0x7f96640fe0e0 2024-02-20T07:57:40.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.506+0000 7f9659ffb700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96500692a0 0x7f965006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:40.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.507+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9660086e90 con 0x7f96640fe0e0 2024-02-20T07:57:40.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.507+0000 7f96692df700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96500692a0 0x7f965006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:40.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.507+0000 7f96692df700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96500692a0 0x7f965006b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9654004890 tx=0x7f9654009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:40.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.511+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9660056c50 con 0x7f96640fe0e0 2024-02-20T07:57:40.587 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:40 smithi055 ceph-mon[29254]: pgmap v181: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 895 B/s wr, 6 op/s 2024-02-20T07:57:40.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:40.753+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9664105a30 con 0x7f96640fe0e0 2024-02-20T07:57:40.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:40 smithi186 ceph-mon[36772]: pgmap v181: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 895 B/s wr, 6 op/s 2024-02-20T07:57:41.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.511+0000 7f9659ffb700 1 -- 172.21.15.55:0/1806787694 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f96600567e0 con 0x7f96640fe0e0 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96500692a0 msgr2=0x7f965006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96500692a0 0x7f965006b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9654004890 tx=0x7f9654009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 msgr2=0x7f966410deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 0x7f966410deb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9660002a50 tx=0x7f96600046b0 comp rx=0 tx=0).stop 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 shutdown_connections 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96640fcef0 0x7f966410d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f96500692a0 0x7f965006b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 --2- 172.21.15.55:0/1806787694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96640fe0e0 0x7f966410deb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:41.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 >> 172.21.15.55:0/1806787694 conn(0x7f96640f8620 msgr2=0x7f96641013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:41.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 shutdown_connections 2024-02-20T07:57:41.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:41.514+0000 7f966b543700 1 -- 172.21.15.55:0/1806787694 wait complete. 2024-02-20T07:57:41.781 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1806787694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-20T07:57:41.781 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:41.781 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:41.782 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:41.782 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.fpwbmz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:41.782 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.fpwbmz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:41.782 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:41 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1806787694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.fpwbmz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi186.fpwbmz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:57:41.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:41 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:42.158 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:42.168 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:42.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-20T07:57:42.477 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:42.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: Deploying daemon mds.cephfs.smithi186.fpwbmz on smithi186 2024-02-20T07:57:42.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-02-20T07:57:42.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-20T07:57:42.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: Cluster is now healthy 2024-02-20T07:57:42.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1806787694' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-20T07:57:42.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:boot 2024-02-20T07:57:42.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 1 up:standby 2024-02-20T07:57:42.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:42 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.sinapd"}]: dispatch 2024-02-20T07:57:42.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: Deploying daemon mds.cephfs.smithi186.fpwbmz on smithi186 2024-02-20T07:57:42.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-02-20T07:57:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-20T07:57:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: Cluster is now healthy 2024-02-20T07:57:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1806787694' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-20T07:57:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:boot 2024-02-20T07:57:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 1 up:standby 2024-02-20T07:57:42.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:42 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.sinapd"}]: dispatch 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/2908026599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 msgr2=0x7fe9f80ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/2908026599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f80ffa80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8004940 tx=0x7fe9e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/2908026599 shutdown_connections 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/2908026599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9f8100050 0x7fe9f81024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/2908026599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f80ffa80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/2908026599 >> 172.21.15.55:0/2908026599 conn(0x7fe9f80f74b0 msgr2=0x7fe9f80f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/2908026599 shutdown_connections 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/2908026599 wait complete. 2024-02-20T07:57:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 Processor -- start 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.071+0000 7fe9fe9f1700 1 -- start start 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9fe9f1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f810d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9fe9f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9f8100050 0x7fe9f810ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9fe9f1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f810e3c0 con 0x7fe9f8066c70 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9fe9f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f810ec60 con 0x7fe9f8100050 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f810d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f810d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57586/0 (socket says 172.21.15.55:57586) 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 -- 172.21.15.55:0/3769885550 learned_addr learned my addr 172.21.15.55:0/3769885550 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f77fe700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9f8100050 0x7fe9f810ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 -- 172.21.15.55:0/3769885550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9f8100050 msgr2=0x7fe9f810ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9f8100050 0x7fe9f810ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 -- 172.21.15.55:0/3769885550 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e8005040 con 0x7fe9f8066c70 2024-02-20T07:57:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f7fff700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f810d880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80093b0 tx=0x7fe9e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e800e040 con 0x7fe9f8066c70 2024-02-20T07:57:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.072+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f810eee0 con 0x7fe9f8066c70 2024-02-20T07:57:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.073+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9e800bcc0 con 0x7fe9f8066c70 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.073+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e801c630 con 0x7fe9f8066c70 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.073+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f810f420 con 0x7fe9f8066c70 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f804bed0 con 0x7fe9f8066c70 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.074+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe9e801c790 con 0x7fe9f8066c70 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.074+0000 7fe9f4ff9700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe9e00692a0 0x7fe9e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.074+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe9e8088430 con 0x7fe9f8066c70 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.074+0000 7fe9f77fe700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe9e00692a0 0x7fe9e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:44.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.075+0000 7fe9f77fe700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe9e00692a0 0x7fe9e006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0041d0 tx=0x7fe9ec0043c0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:44.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.079+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe9e8058340 con 0x7fe9f8066c70 2024-02-20T07:57:44.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:44.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-02-20T07:57:44.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:44.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:44.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:44.327+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe9f810f850 con 0x7fe9f8066c70 2024-02-20T07:57:45.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.072+0000 7fe9f4ff9700 1 -- 172.21.15.55:0/3769885550 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe9e8057ed0 con 0x7fe9f8066c70 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe9e00692a0 msgr2=0x7fe9e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe9e00692a0 0x7fe9e006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0041d0 tx=0x7fe9ec0043c0 comp rx=0 tx=0).stop 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 msgr2=0x7fe9f810d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f810d880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80093b0 tx=0x7fe9e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 shutdown_connections 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fe9e00692a0 0x7fe9e006b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9f8066c70 0x7fe9f810d880 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 --2- 172.21.15.55:0/3769885550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9f8100050 0x7fe9f810ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 >> 172.21.15.55:0/3769885550 conn(0x7fe9f80f74b0 msgr2=0x7fe9f80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 shutdown_connections 2024-02-20T07:57:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:45.074+0000 7fe9fe9f1700 1 -- 172.21.15.55:0/3769885550 wait complete. 2024-02-20T07:57:45.076 INFO:teuthology.orchestra.run.smithi055.stderr:inline data enabled 2024-02-20T07:57:45.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:45 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.186:6826/3352382163,v1:172.21.15.186:6827/3352382163] up:boot 2024-02-20T07:57:45.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:45 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:45.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:45 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi186.fpwbmz"}]: dispatch 2024-02-20T07:57:45.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:45 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3769885550' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-20T07:57:45.367 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:45 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.186:6826/3352382163,v1:172.21.15.186:6827/3352382163] up:boot 2024-02-20T07:57:45.367 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:45 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:45.367 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:45 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi186.fpwbmz"}]: dispatch 2024-02-20T07:57:45.367 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:45 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3769885550' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-20T07:57:46.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-20T07:57:46.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3769885550' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-20T07:57:46.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:standby 2024-02-20T07:57:46.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: Dropping low affinity active daemon mds.cephfs.smithi055.ptjwai in favor of higher affinity standby. 2024-02-20T07:57:46.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: Replacing daemon mds.cephfs.smithi055.ptjwai as rank 0 with standby daemon mds.cephfs.smithi055.sinapd 2024-02-20T07:57:46.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-20T07:57:46.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:46.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: osdmap e46: 6 total, 6 up, 6 in 2024-02-20T07:57:46.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:replay,1=cephfs.smithi186.xntwaj=up:active} 1 up:standby 2024-02-20T07:57:46.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-02-20T07:57:46.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:46.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:46 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:46.202 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:57:46.219 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:57:46.219 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-20T07:57:46.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3769885550' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:standby 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: Dropping low affinity active daemon mds.cephfs.smithi055.ptjwai in favor of higher affinity standby. 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: Replacing daemon mds.cephfs.smithi055.ptjwai as rank 0 with standby daemon mds.cephfs.smithi055.sinapd 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.ptjwai=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:46.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: osdmap e46: 6 total, 6 up, 6 in 2024-02-20T07:57:46.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:replay,1=cephfs.smithi186.xntwaj=up:active} 1 up:standby 2024-02-20T07:57:46.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-02-20T07:57:46.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:46.326 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:46 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:47.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:47 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:resolve 2024-02-20T07:57:47.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:47 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6826/2326607401,v1:172.21.15.55:6827/2326607401] up:boot 2024-02-20T07:57:47.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:47 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] up:active 2024-02-20T07:57:47.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:47 smithi055 ceph-mon[29254]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:resolve,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:47.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:47 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.ptjwai"}]: dispatch 2024-02-20T07:57:47.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:47 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:resolve 2024-02-20T07:57:47.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:47 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6826/2326607401,v1:172.21.15.55:6827/2326607401] up:boot 2024-02-20T07:57:47.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:47 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] up:active 2024-02-20T07:57:47.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:47 smithi186 ceph-mon[36772]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:resolve,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:47.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:47 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.ptjwai"}]: dispatch 2024-02-20T07:57:47.637 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:48.450 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:48 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:reconnect 2024-02-20T07:57:48.450 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:48 smithi055 ceph-mon[29254]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:reconnect,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:48.450 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:48 smithi055 ceph-mon[29254]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 2.1 KiB/s wr, 8 op/s 2024-02-20T07:57:48.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:48 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:reconnect 2024-02-20T07:57:48.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:48 smithi186 ceph-mon[36772]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:reconnect,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:48.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:48 smithi186 ceph-mon[36772]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 2.1 KiB/s wr, 8 op/s 2024-02-20T07:57:49.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:49 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:rejoin 2024-02-20T07:57:49.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:49 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.186:6826/3352382163,v1:172.21.15.186:6827/3352382163] up:standby 2024-02-20T07:57:49.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:49 smithi186 ceph-mon[36772]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:rejoin,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:49.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:49 smithi186 ceph-mon[36772]: daemon mds.cephfs.smithi055.sinapd is now active in filesystem cephfs as rank 0 2024-02-20T07:57:49.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:49 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:49.467 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:49 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:rejoin 2024-02-20T07:57:49.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:49 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.186:6826/3352382163,v1:172.21.15.186:6827/3352382163] up:standby 2024-02-20T07:57:49.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:49 smithi055 ceph-mon[29254]: fsmap cephfs:2/2 {0=cephfs.smithi055.sinapd=up:rejoin,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:49.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:49 smithi055 ceph-mon[29254]: daemon mds.cephfs.smithi055.sinapd is now active in filesystem cephfs as rank 0 2024-02-20T07:57:49.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:49 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 -- 172.21.15.55:0/1076827343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 msgr2=0x7feee0102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1076827343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee0102780 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7feed4004940 tx=0x7feed4009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 -- 172.21.15.55:0/1076827343 shutdown_connections 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1076827343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee0102780 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1076827343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feee00fda80 0x7feee00ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 -- 172.21.15.55:0/1076827343 >> 172.21.15.55:0/1076827343 conn(0x7feee00f74e0 msgr2=0x7feee00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 -- 172.21.15.55:0/1076827343 shutdown_connections 2024-02-20T07:57:49.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 -- 172.21.15.55:0/1076827343 wait complete. 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 Processor -- start 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.795+0000 7feee5c0f700 1 -- start start 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feee5c0f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feee00fda80 0x7feee00fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feee5c0f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee00fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feee5c0f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00fbf90 con 0x7feee01003a0 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feee5c0f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00fc100 con 0x7feee00fda80 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee00fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee00fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57618/0 (socket says 172.21.15.55:57618) 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 -- 172.21.15.55:0/1932484763 learned_addr learned my addr 172.21.15.55:0/1932484763 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedf7fe700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feee00fda80 0x7feee00fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 -- 172.21.15.55:0/1932484763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feee00fda80 msgr2=0x7feee00fd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feee00fda80 0x7feee00fd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:49.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 -- 172.21.15.55:0/1932484763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4005040 con 0x7feee01003a0 2024-02-20T07:57:49.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feedeffd700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee00fb990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7feed40093b0 tx=0x7feed4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:49.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed400e040 con 0x7feee01003a0 2024-02-20T07:57:49.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed400bcc0 con 0x7feee01003a0 2024-02-20T07:57:49.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee00fc380 con 0x7feee01003a0 2024-02-20T07:57:49.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed401c630 con 0x7feee01003a0 2024-02-20T07:57:49.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.796+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee0115970 con 0x7feee01003a0 2024-02-20T07:57:49.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.797+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee004bed0 con 0x7feee01003a0 2024-02-20T07:57:49.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.798+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7feed4013980 con 0x7feee01003a0 2024-02-20T07:57:49.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.798+0000 7feec7fff700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7feec80692a0 0x7feec806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:49.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.798+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7feed40884a0 con 0x7feee01003a0 2024-02-20T07:57:49.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.798+0000 7feedf7fe700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7feec80692a0 0x7feec806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:49.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.799+0000 7feedf7fe700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7feec80692a0 0x7feec806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feee00fd2a0 tx=0x7feed000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:49.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:49.803+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7feed40582b0 con 0x7feee01003a0 2024-02-20T07:57:50.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.058+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feee0105a10 con 0x7feee01003a0 2024-02-20T07:57:50.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.058+0000 7feec7fff700 1 -- 172.21.15.55:0/1932484763 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1819 (secure 0 0 0) 0x7feed4057e40 con 0x7feee01003a0 2024-02-20T07:57:50.060 INFO:teuthology.orchestra.run.smithi055.stdout:e16 2024-02-20T07:57:50.060 INFO:teuthology.orchestra.run.smithi055.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:57:50.060 INFO:teuthology.orchestra.run.smithi055.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:57:50.060 INFO:teuthology.orchestra.run.smithi055.stdout:legacy client fscid: 1 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:Filesystem 'cephfs' (1) 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:fs_name cephfs 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 16 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:created 2024-02-20T07:57:30.395820+0000 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:modified 2024-02-20T07:57:49.107193+0000 2024-02-20T07:57:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:tableserver 0 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:root 0 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:session_timeout 60 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:session_autoclose 300 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:max_file_size 1099511627776 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:required_client_features {} 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure 0 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure_osd_epoch 46 2024-02-20T07:57:50.062 INFO:teuthology.orchestra.run.smithi055.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:max_mds 2 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:in 0,1 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:up {0=14492,1=24253} 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:failed 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:damaged 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:stopped 2024-02-20T07:57:50.063 INFO:teuthology.orchestra.run.smithi055.stdout:data_pools [3] 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:metadata_pool 2 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:inline_data enabled 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:balancer 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:standby_count_wanted 1 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi055.sinapd{0:14492} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi186.xntwaj{1:24253} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.186:6824/1561902443,v1:172.21.15.186:6825/1561902443] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:Standby daemons: 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi055.ptjwai{-1:14500} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.55:6826/2326607401,v1:172.21.15.55:6827/2326607401] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:57:50.064 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi186.fpwbmz{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.186:6826/3352382163,v1:172.21.15.186:6827/3352382163] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:57:50.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7feec80692a0 msgr2=0x7feec806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:50.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7feec80692a0 0x7feec806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feee00fd2a0 tx=0x7feed000b040 comp rx=0 tx=0).stop 2024-02-20T07:57:50.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 msgr2=0x7feee00fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:50.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee00fb990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7feed40093b0 tx=0x7feed4009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:50.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 shutdown_connections 2024-02-20T07:57:50.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feee00fda80 0x7feee00fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:50.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7feec80692a0 0x7feec806b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:50.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.060+0000 7feee5c0f700 1 --2- 172.21.15.55:0/1932484763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feee01003a0 0x7feee00fb990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:50.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.061+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 >> 172.21.15.55:0/1932484763 conn(0x7feee00f74e0 msgr2=0x7feee00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:50.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.061+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 shutdown_connections 2024-02-20T07:57:50.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:50.061+0000 7feee5c0f700 1 -- 172.21.15.55:0/1932484763 wait complete. 2024-02-20T07:57:50.067 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 16 2024-02-20T07:57:50.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-20T07:57:50.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:active 2024-02-20T07:57:50.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.sinapd=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:50.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-20T07:57:50.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:50.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:50.369 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:50.369 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:50.369 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:50.369 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:50 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1932484763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:57:50.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: mds.? [v2:172.21.15.55:6828/3323155944,v1:172.21.15.55:6829/3323155944] up:active 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.sinapd=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:50 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1932484763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:57:50.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-20T07:57:51.396 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:51 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:51.496 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:51 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:57:51.633 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:52.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:52 smithi055 ceph-mon[29254]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-20T07:57:52.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:52 smithi186 ceph-mon[36772]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-20T07:57:53.451 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:53 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:53.451 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:53 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:53.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.552+0000 7f21d35da700 1 -- 172.21.15.55:0/3559961201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fcfc0 msgr2=0x7f21cc0fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:53.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.552+0000 7f21d35da700 1 --2- 172.21.15.55:0/3559961201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fcfc0 0x7f21cc0fd430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f21bc004940 tx=0x7f21bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:53.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- 172.21.15.55:0/3559961201 shutdown_connections 2024-02-20T07:57:53.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 --2- 172.21.15.55:0/3559961201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fcfc0 0x7f21cc0fd430 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 --2- 172.21.15.55:0/3559961201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc0fbe50 0x7f21cc0fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- 172.21.15.55:0/3559961201 >> 172.21.15.55:0/3559961201 conn(0x7f21cc0f7510 msgr2=0x7f21cc0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- 172.21.15.55:0/3559961201 shutdown_connections 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- 172.21.15.55:0/3559961201 wait complete. 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 Processor -- start 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- start start 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 0x7f21cc10d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc0fcfc0 0x7f21cc10de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc10e490 con 0x7f21cc0fbe50 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d35da700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc10ed30 con 0x7f21cc0fcfc0 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d1376700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 0x7f21cc10d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d1376700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 0x7f21cc10d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32850/0 (socket says 172.21.15.55:32850) 2024-02-20T07:57:53.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.553+0000 7f21d1376700 1 -- 172.21.15.55:0/3764488066 learned_addr learned my addr 172.21.15.55:0/3764488066 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d0b75700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc0fcfc0 0x7f21cc10de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d1376700 1 -- 172.21.15.55:0/3764488066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc0fcfc0 msgr2=0x7f21cc10de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d1376700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc0fcfc0 0x7f21cc10de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d1376700 1 -- 172.21.15.55:0/3764488066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc005040 con 0x7f21cc0fbe50 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d1376700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 0x7f21cc10d950 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f21c800c830 tx=0x7f21c8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21c80106f0 con 0x7f21cc0fbe50 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21c8010d20 con 0x7f21cc0fbe50 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc10f010 con 0x7f21cc0fbe50 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21c80113a0 con 0x7f21cc0fbe50 2024-02-20T07:57:53.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.554+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc10f550 con 0x7f21cc0fbe50 2024-02-20T07:57:53.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.555+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc04bed0 con 0x7f21cc0fbe50 2024-02-20T07:57:53.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.555+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f21c8012400 con 0x7f21cc0fbe50 2024-02-20T07:57:53.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.556+0000 7f21c1ffb700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f21b80692a0 0x7f21b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:53.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.556+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f21c80870b0 con 0x7f21cc0fbe50 2024-02-20T07:57:53.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.556+0000 7f21d0b75700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f21b80692a0 0x7f21b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:53.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.556+0000 7f21d0b75700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f21b80692a0 0x7f21b806b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00b3b0 tx=0x7f21bc009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:53.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.560+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f21c8056ec0 con 0x7f21cc0fbe50 2024-02-20T07:57:53.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:53 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:53.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:53 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:53.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.796+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21cc104940 con 0x7f21cc0fbe50 2024-02-20T07:57:53.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.797+0000 7f21c1ffb700 1 -- 172.21.15.55:0/3764488066 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4709 (secure 0 0 0) 0x7f21c8056a50 con 0x7f21cc0fbe50 2024-02-20T07:57:53.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f21b80692a0 msgr2=0x7f21b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f21b80692a0 0x7f21b806b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00b3b0 tx=0x7f21bc009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 msgr2=0x7f21cc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 0x7f21cc10d950 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f21c800c830 tx=0x7f21c8004470 comp rx=0 tx=0).stop 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 shutdown_connections 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f21b80692a0 0x7f21b806b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21cc0fbe50 0x7f21cc10d950 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:53.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 --2- 172.21.15.55:0/3764488066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc0fcfc0 0x7f21cc10de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:53.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 >> 172.21.15.55:0/3764488066 conn(0x7f21cc0f7510 msgr2=0x7f21cc065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:53.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 shutdown_connections 2024-02-20T07:57:53.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:53.799+0000 7f21d35da700 1 -- 172.21.15.55:0/3764488066 wait complete. 2024-02-20T07:57:53.804 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 16 2024-02-20T07:57:53.810 INFO:teuthology.orchestra.run.smithi055.stdout:true 2024-02-20T07:57:54.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:54 smithi055 ceph-mon[29254]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-20T07:57:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:54 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3764488066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:57:54.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:54 smithi186 ceph-mon[36772]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-20T07:57:54.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:54 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3764488066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:57:54.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-20T07:57:55.380 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:57:56.459 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:56 smithi055 ceph-mon[29254]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-20T07:57:56.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:56 smithi186 ceph-mon[36772]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-20T07:57:58.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 -- 172.21.15.55:0/481755751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fcfb0 msgr2=0x7f64740fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:58.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 --2- 172.21.15.55:0/481755751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fcfb0 0x7f64740fd420 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6464004940 tx=0x7f6464009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:58.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 -- 172.21.15.55:0/481755751 shutdown_connections 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 --2- 172.21.15.55:0/481755751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fcfb0 0x7f64740fd420 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 --2- 172.21.15.55:0/481755751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64740fbe40 0x7f64740fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 -- 172.21.15.55:0/481755751 >> 172.21.15.55:0/481755751 conn(0x7f64740f7520 msgr2=0x7f64740f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 -- 172.21.15.55:0/481755751 shutdown_connections 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 -- 172.21.15.55:0/481755751 wait complete. 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 Processor -- start 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 -- start start 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.085+0000 7f647b64b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 0x7f647410b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f647b64b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64740fcfb0 0x7f647410bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f647b64b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647410c180 con 0x7f64740fbe40 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f647b64b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647410ca20 con 0x7f64740fcfb0 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 0x7f647410b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 0x7f647410b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32862/0 (socket says 172.21.15.55:32862) 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 -- 172.21.15.55:0/1354624136 learned_addr learned my addr 172.21.15.55:0/1354624136 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f6478be6700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64740fcfb0 0x7f647410bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 -- 172.21.15.55:0/1354624136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64740fcfb0 msgr2=0x7f647410bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64740fcfb0 0x7f647410bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 -- 172.21.15.55:0/1354624136 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464005040 con 0x7f64740fbe40 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f64793e7700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 0x7f647410b640 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f647000eb40 tx=0x7f647000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6470007cc0 con 0x7f64740fbe40 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647410cca0 con 0x7f64740fbe40 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f647000f040 con 0x7f64740fbe40 2024-02-20T07:57:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.086+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647000ca10 con 0x7f64740fbe40 2024-02-20T07:57:58.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.087+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647410d1e0 con 0x7f64740fbe40 2024-02-20T07:57:58.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.088+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f647000cc80 con 0x7f64740fbe40 2024-02-20T07:57:58.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.088+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647404bed0 con 0x7f64740fbe40 2024-02-20T07:57:58.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.088+0000 7f6469ffb700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64600692a0 0x7f646006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:58.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.088+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6470086ce0 con 0x7f64740fbe40 2024-02-20T07:57:58.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.088+0000 7f6478be6700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64600692a0 0x7f646006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:58.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.089+0000 7f6478be6700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64600692a0 0x7f646006b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64640118f0 tx=0x7f6464009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:58.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.093+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f64700569d0 con 0x7f64740fbe40 2024-02-20T07:57:58.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.399+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f64741037d0 con 0x7f64740fbe40 2024-02-20T07:57:58.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.399+0000 7f6469ffb700 1 -- 172.21.15.55:0/1354624136 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+99 (secure 0 0 0) 0x7f6470056560 con 0x7f64740fbe40 2024-02-20T07:57:58.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64600692a0 msgr2=0x7f646006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:58.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64600692a0 0x7f646006b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64640118f0 tx=0x7f6464009e30 comp rx=0 tx=0).stop 2024-02-20T07:57:58.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 msgr2=0x7f647410b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 0x7f647410b640 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f647000eb40 tx=0x7f647000b040 comp rx=0 tx=0).stop 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 shutdown_connections 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64600692a0 0x7f646006b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64740fbe40 0x7f647410b640 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 --2- 172.21.15.55:0/1354624136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64740fcfb0 0x7f647410bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 >> 172.21.15.55:0/1354624136 conn(0x7f64740f7520 msgr2=0x7f6474100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 shutdown_connections 2024-02-20T07:57:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:58.401+0000 7f647b64b700 1 -- 172.21.15.55:0/1354624136 wait complete. 2024-02-20T07:57:58.412 INFO:teuthology.orchestra.run.smithi055.stdout:true 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:58.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-20T07:57:58.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:57:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:57:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:57:58.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:57:59.034 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-20T07:57:59.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 -- 172.21.15.55:0/3725205200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fd530 msgr2=0x7f34080fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 --2- 172.21.15.55:0/3725205200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fd530 0x7f34080fd9a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009c50 tx=0x7f33f8004d10 comp rx=0 tx=0).stop 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 -- 172.21.15.55:0/3725205200 shutdown_connections 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 --2- 172.21.15.55:0/3725205200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fd530 0x7f34080fd9a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 --2- 172.21.15.55:0/3725205200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fc260 0x7f34080fc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 -- 172.21.15.55:0/3725205200 >> 172.21.15.55:0/3725205200 conn(0x7f34080f7a00 msgr2=0x7f34080f9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 -- 172.21.15.55:0/3725205200 shutdown_connections 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 -- 172.21.15.55:0/3725205200 wait complete. 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 Processor -- start 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.164+0000 7f340ef1d640 1 -- start start 2024-02-20T07:57:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ef1d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 0x7f340810cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ef1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fd530 0x7f340810d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ef1d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340810d6b0 con 0x7f34080fc260 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ef1d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3408112100 con 0x7f34080fd530 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 0x7f340810cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 0x7f340810cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32874/0 (socket says 172.21.15.55:32874) 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 -- 172.21.15.55:0/2008561705 learned_addr learned my addr 172.21.15.55:0/2008561705 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 -- 172.21.15.55:0/2008561705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fd530 msgr2=0x7f340810d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f3407fff640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fd530 0x7f340810d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fd530 0x7f340810d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 -- 172.21.15.55:0/2008561705 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80098b0 con 0x7f34080fc260 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f3407fff640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fd530 0x7f340810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ccb9640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 0x7f340810cb20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f340000c830 tx=0x7f3400004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340000a850 con 0x7f34080fc260 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f340000ae80 con 0x7f34080fc260 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34081122a0 con 0x7f34080fc260 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3400012ea0 con 0x7f34080fc260 2024-02-20T07:57:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.165+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3408112810 con 0x7f34080fc260 2024-02-20T07:57:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.166+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3408104db0 con 0x7f34080fc260 2024-02-20T07:57:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.167+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3400012400 con 0x7f34080fc260 2024-02-20T07:57:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.167+0000 7f34057fa640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f33f4069210 0x7f33f406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.167+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3400087450 con 0x7f34080fc260 2024-02-20T07:57:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.167+0000 7f3407fff640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f33f4069210 0x7f33f406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.168+0000 7f3407fff640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f33f4069210 0x7f33f406b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f33f802c120 tx=0x7f33f802e040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:59.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.172+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3400053730 con 0x7f34080fc260 2024-02-20T07:57:59.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.345+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3408112f30 con 0x7f34080fc260 2024-02-20T07:57:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.346+0000 7f34057fa640 1 -- 172.21.15.55:0/2008561705 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14385 (secure 0 0 0) 0x7f34000126b0 con 0x7f34080fc260 2024-02-20T07:57:59.347 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:57:59.347 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":46,"fsid":"9a1162a4-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:50:23.257218+0000","modified":"2024-02-20T07:57:45.072998+0000","last_up_change":"2024-02-20T07:56:11.165108+0000","last_in_change":"2024-02-20T07:55:52.356730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:54:59.573551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-20T07:57:29.277692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-20T07:57:29.619471+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16d58e18-643f-4fc7-bb34-76cb0ba30b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6803","nonce":3812016597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6805","nonce":3812016597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6809","nonce":3812016597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3812016597},{"type":"v1","addr":"172.21.15.55:6807","nonce":3812016597}]},"public_addr":"172.21.15.55:6803/3812016597","cluster_addr":"172.21.15.55:6805/3812016597","heartbeat_back_addr":"172.21.15.55:6809/3812016597","heartbeat_front_addr":"172.21.15.55:6807/3812016597","state":["exists","up"]},{"osd":1,"uuid":"089fab11-e47a-425b-83d8-10bc9dce41e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6811","nonce":4284510212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6813","nonce":4284510212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6817","nonce":4284510212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4284510212},{"type":"v1","addr":"172.21.15.55:6815","nonce":4284510212}]},"public_addr":"172.21.15.55:6811/4284510212","cluster_addr":"172.21.15.55:6813/4284510212","heartbeat_back_addr":"172.21.15.55:6817/4284510212","heartbeat_front_addr":"172.21.15.55:6815/4284510212","state":["exists","up"]},{"osd":2,"uuid":"ecf84045-e8d4-4ef5-bbdb-c95d21ce4647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6819","nonce":138638858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6821","nonce":138638858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6825","nonce":138638858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":138638858},{"type":"v1","addr":"172.21.15.55:6823","nonce":138638858}]},"public_addr":"172.21.15.55:6819/138638858","cluster_addr":"172.21.15.55:6821/138638858","heartbeat_back_addr":"172.21.15.55:6825/138638858","heartbeat_front_addr":"172.21.15.55:6823/138638858","state":["exists","up"]},{"osd":3,"uuid":"29312f2b-ee24-4c2b-a517-10b3a1740ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6801","nonce":3751122696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6803","nonce":3751122696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6807","nonce":3751122696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3751122696},{"type":"v1","addr":"172.21.15.186:6805","nonce":3751122696}]},"public_addr":"172.21.15.186:6801/3751122696","cluster_addr":"172.21.15.186:6803/3751122696","heartbeat_back_addr":"172.21.15.186:6807/3751122696","heartbeat_front_addr":"172.21.15.186:6805/3751122696","state":["exists","up"]},{"osd":4,"uuid":"c15e91be-1400-4d62-9eb9-1f761108b4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6809","nonce":710072980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6811","nonce":710072980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6815","nonce":710072980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":710072980},{"type":"v1","addr":"172.21.15.186:6813","nonce":710072980}]},"public_addr":"172.21.15.186:6809/710072980","cluster_addr":"172.21.15.186:6811/710072980","heartbeat_back_addr":"172.21.15.186:6815/710072980","heartbeat_front_addr":"172.21.15.186:6813/710072980","state":["exists","up"]},{"osd":5,"uuid":"9a479e8d-266a-489a-821b-962e81638bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6817","nonce":3358744054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6819","nonce":3358744054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6823","nonce":3358744054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3358744054},{"type":"v1","addr":"172.21.15.186:6821","nonce":3358744054}]},"public_addr":"172.21.15.186:6817/3358744054","cluster_addr":"172.21.15.186:6819/3358744054","heartbeat_back_addr":"172.21.15.186:6823/3358744054","heartbeat_front_addr":"172.21.15.186:6821/3358744054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:57.185645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:28.096711+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:54:56.659378+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:19.780229+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:55:44.427615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:56:09.309844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6801/117641079":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6801/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/736324230":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6800/4027903862":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/4098881425":"2024-02-21T07:51:24.632021+0000","172.21.15.55:6800/117641079":"2024-02-21T07:50:53.618922+0000","172.21.15.55:0/1844035838":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2074396018":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6827/3290843304":"2024-02-21T07:57:45.072974+0000","172.21.15.55:6801/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/3177458092":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/1896683634":"2024-02-21T07:51:24.632021+0000","172.21.15.55:0/2770110765":"2024-02-21T07:52:43.239066+0000","172.21.15.55:6800/2284801197":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2141854148":"2024-02-21T07:52:43.239066+0000","172.21.15.55:0/2307704898":"2024-02-21T07:50:53.618922+0000","172.21.15.55:6826/3290843304":"2024-02-21T07:57:45.072974+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:57:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f33f4069210 msgr2=0x7f33f406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f33f4069210 0x7f33f406b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f33f802c120 tx=0x7f33f802e040 comp rx=0 tx=0).stop 2024-02-20T07:57:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 msgr2=0x7f340810cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 0x7f340810cb20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f340000c830 tx=0x7f3400004430 comp rx=0 tx=0).stop 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 shutdown_connections 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f33f4069210 0x7f33f406b6c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34080fd530 0x7f340810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 --2- 172.21.15.55:0/2008561705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34080fc260 0x7f340810cb20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 >> 172.21.15.55:0/2008561705 conn(0x7f34080f7a00 msgr2=0x7f3408100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 shutdown_connections 2024-02-20T07:57:59.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.348+0000 7f340ef1d640 1 -- 172.21.15.55:0/2008561705 wait complete. 2024-02-20T07:57:59.360 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-20T07:54:59.573551+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-20T07:57:29.277692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-20T07:57:29.619471+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-20T07:57:59.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 -- 172.21.15.55:0/3139278669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fb180 msgr2=0x7f4af80fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 --2- 172.21.15.55:0/3139278669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fb180 0x7f4af80fb590 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80048d0 tx=0x7f4ae8009da0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 -- 172.21.15.55:0/3139278669 shutdown_connections 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 --2- 172.21.15.55:0/3139278669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4af80fc450 0x7f4af80fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 --2- 172.21.15.55:0/3139278669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fb180 0x7f4af80fb590 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 -- 172.21.15.55:0/3139278669 >> 172.21.15.55:0/3139278669 conn(0x7f4af80f6960 msgr2=0x7f4af80f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 -- 172.21.15.55:0/3139278669 shutdown_connections 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 -- 172.21.15.55:0/3139278669 wait complete. 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 Processor -- start 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.461+0000 7f4afd71d640 1 -- start start 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4afd71d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4af80fb180 0x7f4af810cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4afd71d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 0x7f4af810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4afd71d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af810d6e0 con 0x7f4af80fc450 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4afd71d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8112130 con 0x7f4af80fb180 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 0x7f4af810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 0x7f4af810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32902/0 (socket says 172.21.15.55:32902) 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 -- 172.21.15.55:0/1323082158 learned_addr learned my addr 172.21.15.55:0/1323082158 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4af6ffd640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4af80fb180 0x7f4af810cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 -- 172.21.15.55:0/1323082158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4af80fb180 msgr2=0x7f4af810cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4af80fb180 0x7f4af810cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 -- 172.21.15.55:0/1323082158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8005040 con 0x7f4af80fc450 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aeffff640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 0x7f4af810d090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0002a50 tx=0x7f4ae0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae0007c30 con 0x7f4af80fc450 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae000f040 con 0x7f4af80fc450 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af81122d0 con 0x7f4af80fc450 2024-02-20T07:57:59.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae000db50 con 0x7f4af80fc450 2024-02-20T07:57:59.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.462+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af81127c0 con 0x7f4af80fc450 2024-02-20T07:57:59.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.463+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af8103cf0 con 0x7f4af80fc450 2024-02-20T07:57:59.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.464+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4ae001c400 con 0x7f4af80fc450 2024-02-20T07:57:59.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.464+0000 7f4aef7fe640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ae4069210 0x7f4ae406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.464+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4ae0086a20 con 0x7f4af80fc450 2024-02-20T07:57:59.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.464+0000 7f4af6ffd640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ae4069210 0x7f4ae406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.465+0000 7f4af6ffd640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ae4069210 0x7f4ae406b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8000c00 tx=0x7f4ae8009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:59.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.468+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4ae004cbd0 con 0x7f4af80fc450 2024-02-20T07:57:59.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.639+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4af81129e0 con 0x7f4af80fc450 2024-02-20T07:57:59.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.639+0000 7f4aef7fe640 1 -- 172.21.15.55:0/1323082158 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ae0052d30 con 0x7f4af80fc450 2024-02-20T07:57:59.640 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ae4069210 msgr2=0x7f4ae406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ae4069210 0x7f4ae406b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8000c00 tx=0x7f4ae8009da0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 msgr2=0x7f4af810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 0x7f4af810d090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0002a50 tx=0x7f4ae0004600 comp rx=0 tx=0).stop 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 shutdown_connections 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f4ae4069210 0x7f4ae406b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4af80fc450 0x7f4af810d090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 --2- 172.21.15.55:0/1323082158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4af80fb180 0x7f4af810cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 >> 172.21.15.55:0/1323082158 conn(0x7f4af80f6960 msgr2=0x7f4af80656d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 shutdown_connections 2024-02-20T07:57:59.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.641+0000 7f4afd71d640 1 -- 172.21.15.55:0/1323082158 wait complete. 2024-02-20T07:57:59.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:59 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1354624136' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-20T07:57:59.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:57:59 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2008561705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:57:59.653 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 -- 172.21.15.55:0/3771189064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 msgr2=0x7f777c0fda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 --2- 172.21.15.55:0/3771189064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c0fda50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f776c0048d0 tx=0x7f776c009da0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 -- 172.21.15.55:0/3771189064 shutdown_connections 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 --2- 172.21.15.55:0/3771189064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c0fda50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 --2- 172.21.15.55:0/3771189064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 0x7f777c0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 -- 172.21.15.55:0/3771189064 >> 172.21.15.55:0/3771189064 conn(0x7f777c0f5ff0 msgr2=0x7f777c0f8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 -- 172.21.15.55:0/3771189064 shutdown_connections 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 -- 172.21.15.55:0/3771189064 wait complete. 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 Processor -- start 2024-02-20T07:57:59.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.753+0000 7f778399d640 1 -- start start 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f778399d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 0x7f777c10cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f778399d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c10d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f778399d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c10d720 con 0x7f777c0fd5e0 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f778399d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c112170 con 0x7f777c0fa5d0 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c10d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c10d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32920/0 (socket says 172.21.15.55:32920) 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 -- 172.21.15.55:0/3591495961 learned_addr learned my addr 172.21.15.55:0/3591495961 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7781739640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 0x7f777c10cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 -- 172.21.15.55:0/3591495961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 msgr2=0x7f777c10cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 0x7f777c10cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 -- 172.21.15.55:0/3591495961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c005040 con 0x7f777c0fd5e0 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7781739640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 0x7f777c10cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7780f38640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c10d0d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f776c034040 tx=0x7f776c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:59.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f776c009590 con 0x7f777c0fd5e0 2024-02-20T07:57:59.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f776c0096f0 con 0x7f777c0fd5e0 2024-02-20T07:57:59.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c1122b0 con 0x7f777c0fd5e0 2024-02-20T07:57:59.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.754+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f776c03b6b0 con 0x7f777c0fd5e0 2024-02-20T07:57:59.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.755+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c1127a0 con 0x7f777c0fd5e0 2024-02-20T07:57:59.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.755+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c104e50 con 0x7f777c0fd5e0 2024-02-20T07:57:59.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.756+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f776c02f070 con 0x7f777c0fd5e0 2024-02-20T07:57:59.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.756+0000 7f7771ffb640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7768069210 0x7f776806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:57:59.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.756+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f776c0a9a30 con 0x7f777c0fd5e0 2024-02-20T07:57:59.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.756+0000 7f7781739640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7768069210 0x7f776806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:57:59.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.757+0000 7f7781739640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7768069210 0x7f776806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f777c065410 tx=0x7f7778009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:57:59.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.761+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f776c075bf0 con 0x7f777c0fd5e0 2024-02-20T07:57:59.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:59 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1354624136' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-20T07:57:59.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:57:59 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2008561705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:57:59.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.930+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f777c112a80 con 0x7f777c0fd5e0 2024-02-20T07:57:59.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.931+0000 7f7771ffb640 1 -- 172.21.15.55:0/3591495961 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f776c079210 con 0x7f777c0fd5e0 2024-02-20T07:57:59.932 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-02-20T07:57:59.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7768069210 msgr2=0x7f776806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7768069210 0x7f776806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f777c065410 tx=0x7f7778009040 comp rx=0 tx=0).stop 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 msgr2=0x7f777c10d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c10d0d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f776c034040 tx=0x7f776c009da0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 shutdown_connections 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7768069210 0x7f776806b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f777c0fd5e0 0x7f777c10d0d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 --2- 172.21.15.55:0/3591495961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777c0fa5d0 0x7f777c10cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 >> 172.21.15.55:0/3591495961 conn(0x7f777c0f5ff0 msgr2=0x7f777c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 shutdown_connections 2024-02-20T07:57:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:57:59.933+0000 7f778399d640 1 -- 172.21.15.55:0/3591495961 wait complete. 2024-02-20T07:57:59.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-20T07:58:00.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- 172.21.15.55:0/1320525227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04280ff430 msgr2=0x7f0428101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 --2- 172.21.15.55:0/1320525227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04280ff430 0x7f0428101810 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f041c0048d0 tx=0x7f041c009da0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- 172.21.15.55:0/1320525227 shutdown_connections 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 --2- 172.21.15.55:0/1320525227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04280ff430 0x7f0428101810 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 --2- 172.21.15.55:0/1320525227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0428066310 0x7f04280feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- 172.21.15.55:0/1320525227 >> 172.21.15.55:0/1320525227 conn(0x7f04280f6960 msgr2=0x7f04280f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- 172.21.15.55:0/1320525227 shutdown_connections 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- 172.21.15.55:0/1320525227 wait complete. 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 Processor -- start 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- start start 2024-02-20T07:58:00.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 0x7f042810cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04280ff430 0x7f042810d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042810d6f0 con 0x7f0428066310 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f042db9d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0428112140 con 0x7f04280ff430 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.054+0000 7f04277fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 0x7f042810cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f04277fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 0x7f042810cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32934/0 (socket says 172.21.15.55:32934) 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f04277fe640 1 -- 172.21.15.55:0/561412448 learned_addr learned my addr 172.21.15.55:0/561412448 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f0426ffd640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04280ff430 0x7f042810d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f04277fe640 1 -- 172.21.15.55:0/561412448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04280ff430 msgr2=0x7f042810d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f04277fe640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04280ff430 0x7f042810d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f04277fe640 1 -- 172.21.15.55:0/561412448 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c005040 con 0x7f0428066310 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f04277fe640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 0x7f042810cb60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0418002a80 tx=0x7f0418004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0418007c30 con 0x7f0428066310 2024-02-20T07:58:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f041800f040 con 0x7f0428066310 2024-02-20T07:58:00.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04281122e0 con 0x7f0428066310 2024-02-20T07:58:00.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0418009650 con 0x7f0428066310 2024-02-20T07:58:00.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.055+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04281127d0 con 0x7f0428066310 2024-02-20T07:58:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.056+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0428106d90 con 0x7f0428066310 2024-02-20T07:58:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.056+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f041800d6b0 con 0x7f0428066310 2024-02-20T07:58:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.057+0000 7f040ffff640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0410069210 0x7f041006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.057+0000 7f0426ffd640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0410069210 0x7f041006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.057+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0418087100 con 0x7f0428066310 2024-02-20T07:58:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.057+0000 7f0426ffd640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0410069210 0x7f041006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f041c009fa0 tx=0x7f041c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:00.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.061+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0418053360 con 0x7f0428066310 2024-02-20T07:58:00.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.233+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f042802a8e0 con 0x7f0428066310 2024-02-20T07:58:00.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.233+0000 7f040ffff640 1 -- 172.21.15.55:0/561412448 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0418056980 con 0x7f0428066310 2024-02-20T07:58:00.235 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0410069210 msgr2=0x7f041006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0410069210 0x7f041006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f041c009fa0 tx=0x7f041c009da0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 msgr2=0x7f042810cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 0x7f042810cb60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0418002a80 tx=0x7f0418004570 comp rx=0 tx=0).stop 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 shutdown_connections 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0410069210 0x7f041006b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04280ff430 0x7f042810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 --2- 172.21.15.55:0/561412448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428066310 0x7f042810cb60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 >> 172.21.15.55:0/561412448 conn(0x7f04280f6960 msgr2=0x7f04280f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:00.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 shutdown_connections 2024-02-20T07:58:00.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.235+0000 7f042db9d640 1 -- 172.21.15.55:0/561412448 wait complete. 2024-02-20T07:58:00.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-20T07:58:00.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 -- 172.21.15.55:0/899653014 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 msgr2=0x7fb5180fc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 --2- 172.21.15.55:0/899653014 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb5180fc6a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb5080048d0 tx=0x7fb508009da0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 -- 172.21.15.55:0/899653014 shutdown_connections 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 --2- 172.21.15.55:0/899653014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 0x7fb5180fd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 --2- 172.21.15.55:0/899653014 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb5180fc6a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 -- 172.21.15.55:0/899653014 >> 172.21.15.55:0/899653014 conn(0x7fb5180f7a30 msgr2=0x7fb5180f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 -- 172.21.15.55:0/899653014 shutdown_connections 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 -- 172.21.15.55:0/899653014 wait complete. 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 Processor -- start 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.345+0000 7fb51db9d640 1 -- start start 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb51db9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb51810cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb51db9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 0x7fb51810d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:00.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb51db9d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51810d6b0 con 0x7fb5180fc290 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb51db9d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb518112100 con 0x7fb5180fd560 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb51810cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb51810cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32952/0 (socket says 172.21.15.55:32952) 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 -- 172.21.15.55:0/371778614 learned_addr learned my addr 172.21.15.55:0/371778614 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb516ffd640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 0x7fb51810d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 -- 172.21.15.55:0/371778614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 msgr2=0x7fb51810d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 0x7fb51810d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 -- 172.21.15.55:0/371778614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508005040 con 0x7fb5180fc290 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb516ffd640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 0x7fb51810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb5177fe640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb51810cb20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb508007d20 tx=0x7fb508009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb508009590 con 0x7fb5180fc290 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5080096f0 con 0x7fb5180fc290 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb518112240 con 0x7fb5180fc290 2024-02-20T07:58:00.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb50803b6d0 con 0x7fb5180fc290 2024-02-20T07:58:00.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.346+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb518112730 con 0x7fb5180fc290 2024-02-20T07:58:00.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.347+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb518104e00 con 0x7fb5180fc290 2024-02-20T07:58:00.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.348+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb50802f070 con 0x7fb5180fc290 2024-02-20T07:58:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.348+0000 7fb4fffff640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb500069210 0x7fb50006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.348+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb5080a9990 con 0x7fb5180fc290 2024-02-20T07:58:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.348+0000 7fb516ffd640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb500069210 0x7fb50006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.348+0000 7fb516ffd640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb500069210 0x7fb50006b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb51810e070 tx=0x7fb50c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:00.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.352+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb508075bf0 con 0x7fb5180fc290 2024-02-20T07:58:00.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.562+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb518112a10 con 0x7fb5180fc290 2024-02-20T07:58:00.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.563+0000 7fb4fffff640 1 -- 172.21.15.55:0/371778614 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4709 (secure 0 0 0) 0x7fb508079210 con 0x7fb5180fc290 2024-02-20T07:58:00.564 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-02-20T07:58:00.564 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14500,"name":"cephfs.smithi055.ptjwai","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6827/2326607401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2326607401},{"type":"v1","addr":"172.21.15.55:6827","nonce":2326607401}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24269,"name":"cephfs.smithi186.fpwbmz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.186:6827/3352382163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3352382163},{"type":"v1","addr":"172.21.15.186:6827","nonce":3352382163}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-20T07:57:30.395820+0000","modified":"2024-02-20T07:57:49.107193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14492,"mds_1":24253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14492":{"gid":14492,"name":"cephfs.smithi055.sinapd","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.55:6829/3323155944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3323155944},{"type":"v1","addr":"172.21.15.55:6829","nonce":3323155944}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24253":{"gid":24253,"name":"cephfs.smithi186.xntwaj","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.186:6825/1561902443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1561902443},{"type":"v1","addr":"172.21.15.186:6825","nonce":1561902443}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-20T07:58:00.564 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 16 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb500069210 msgr2=0x7fb50006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb500069210 0x7fb50006b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb51810e070 tx=0x7fb50c009040 comp rx=0 tx=0).stop 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 msgr2=0x7fb51810cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb51810cb20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb508007d20 tx=0x7fb508009da0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 shutdown_connections 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb500069210 0x7fb50006b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5180fd560 0x7fb51810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 --2- 172.21.15.55:0/371778614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5180fc290 0x7fb51810cb20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.564+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 >> 172.21.15.55:0/371778614 conn(0x7fb5180f7a30 msgr2=0x7fb518100790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.565+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 shutdown_connections 2024-02-20T07:58:00.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:00.565+0000 7fb51db9d640 1 -- 172.21.15.55:0/371778614 wait complete. 2024-02-20T07:58:00.575 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:00 smithi055 ceph-mon[29254]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-20T07:58:00.575 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:00 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1323082158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:58:00.575 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:00 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/3591495961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-20T07:58:00.575 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:00 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/561412448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-20T07:58:00.576 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 2, 'flags': 18} 2024-02-20T07:58:00.577 INFO:teuthology.run_tasks:Running task kclient... 2024-02-20T07:58:00.900 INFO:tasks.kclient:Mounting kernel clients... 2024-02-20T07:58:00.900 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-20T07:58:00.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:00.901 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-02-20T07:58:00.901 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:00 smithi186 ceph-mon[36772]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-20T07:58:00.901 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:00 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1323082158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:58:00.902 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:00 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/3591495961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-20T07:58:00.902 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:00 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/561412448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-20T07:58:00.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:00.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link delete ceph-brx 2024-02-20T07:58:00.989 INFO:teuthology.orchestra.run.smithi055.stderr:Cannot find device "ceph-brx" 2024-02-20T07:58:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:58:00.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:00.991 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-02-20T07:58:01.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:01.011 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-02-20T07:58:01.088 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2024-02-20T07:58:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:58:01.090 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-20T07:58:01.090 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-20T07:58:01.090 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:01.090 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-20T07:58:01.357 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:01.694 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:01 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/371778614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:58:01.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:01 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/371778614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:58:02.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:02 smithi186 ceph-mon[36772]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-20T07:58:02.630 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:02 smithi055 ceph-mon[29254]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 -- 172.21.15.55:0/846062495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 msgr2=0x7ffbbc0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 --2- 172.21.15.55:0/846062495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc0fbe70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffbac004940 tx=0x7ffbac009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 -- 172.21.15.55:0/846062495 shutdown_connections 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 --2- 172.21.15.55:0/846062495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc0fbe70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 --2- 172.21.15.55:0/846062495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbbc101e60 0x7ffbbc102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 -- 172.21.15.55:0/846062495 >> 172.21.15.55:0/846062495 conn(0x7ffbbc0f7a10 msgr2=0x7ffbbc0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 -- 172.21.15.55:0/846062495 shutdown_connections 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.995+0000 7ffbc1127700 1 -- 172.21.15.55:0/846062495 wait complete. 2024-02-20T07:58:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbc1127700 1 Processor -- start 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbc1127700 1 -- start start 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbc1127700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbbc101e60 0x7ffbbc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbc1127700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbc1127700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc114670 con 0x7ffbbc102800 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbc1127700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc10d9a0 con 0x7ffbbc101e60 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbba59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbba59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43142/0 (socket says 172.21.15.55:43142) 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbba59c700 1 -- 172.21.15.55:0/1187846617 learned_addr learned my addr 172.21.15.55:0/1187846617 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.996+0000 7ffbbad9d700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbbc101e60 0x7ffbbc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffbba59c700 1 -- 172.21.15.55:0/1187846617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbbc101e60 msgr2=0x7ffbbc113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffbba59c700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbbc101e60 0x7ffbbc113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffbba59c700 1 -- 172.21.15.55:0/1187846617 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac005040 con 0x7ffbbc102800 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffbba59c700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc114020 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0090d0 tx=0x7ffbac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbac00e040 con 0x7ffbbc102800 2024-02-20T07:58:02.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbac009430 con 0x7ffbbc102800 2024-02-20T07:58:03.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc10dc20 con 0x7ffbbc102800 2024-02-20T07:58:03.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbac01c3f0 con 0x7ffbbc102800 2024-02-20T07:58:03.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.997+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc10e160 con 0x7ffbbc102800 2024-02-20T07:58:03.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.998+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc0fb610 con 0x7ffbbc102800 2024-02-20T07:58:03.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.998+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ffbac00bbf0 con 0x7ffbbc102800 2024-02-20T07:58:03.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.999+0000 7ffba37fe700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ffba40692a0 0x7ffba406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:03.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.999+0000 7ffbbad9d700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ffba40692a0 0x7ffba406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:03.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.999+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffbac0884c0 con 0x7ffbbc102800 2024-02-20T07:58:03.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:02.999+0000 7ffbbad9d700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ffba40692a0 0x7ffba406b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0004eb0 tx=0x7ffbb000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:03.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.003+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ffbac058230 con 0x7ffbbc102800 2024-02-20T07:58:03.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.245+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffbbc108f70 con 0x7ffbbc102800 2024-02-20T07:58:03.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.246+0000 7ffba37fe700 1 -- 172.21.15.55:0/1187846617 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7ffbac057dc0 con 0x7ffbbc102800 2024-02-20T07:58:03.247 INFO:teuthology.orchestra.run.smithi055.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ffba40692a0 msgr2=0x7ffba406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ffba40692a0 0x7ffba406b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0004eb0 tx=0x7ffbb000b040 comp rx=0 tx=0).stop 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 msgr2=0x7ffbbc114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc114020 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0090d0 tx=0x7ffbac009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 shutdown_connections 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbbc101e60 0x7ffbbc113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ffba40692a0 0x7ffba406b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 --2- 172.21.15.55:0/1187846617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc102800 0x7ffbbc114020 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 >> 172.21.15.55:0/1187846617 conn(0x7ffbbc0f7a10 msgr2=0x7ffbbc065ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:03.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 shutdown_connections 2024-02-20T07:58:03.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:03.248+0000 7ffbc1127700 1 -- 172.21.15.55:0/1187846617 wait complete. 2024-02-20T07:58:03.856 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-20T07:58:03.856 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-20T07:58:03.856 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-02-20T07:58:03.856 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-20T07:58:03.856 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:03.857 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-20T07:58:03.857 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-20T07:58:03.857 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-20T07:58:03.857 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-20T07:58:03.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:03.857 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:58:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42236sec preferred_lft 42236sec 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:03.875 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-02-20T07:58:03.876 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-20T07:58:03.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:03.876 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2024-02-20T07:58:03.876 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-02-20T07:58:03.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add name ceph-brx type bridge 2024-02-20T07:58:03.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr flush dev ceph-brx 2024-02-20T07:58:03.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set ceph-brx up 2024-02-20T07:58:03.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-20T07:58:03.877 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-02-20T07:58:04.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.032 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-20T07:58:04.059 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-02-20T07:58:04.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.061 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-02-20T07:58:04.134 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-02-20T07:58:04.134 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-20T07:58:04.134 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-20T07:58:04.134 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-20T07:58:04.134 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-20T07:58:04.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.135 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2024-02-20T07:58:04.135 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-02-20T07:58:04.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-20T07:58:04.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-20T07:58:04.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-20T07:58:04.135 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-02-20T07:58:04.197 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:04 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1187846617' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:58:04.197 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:04 smithi055 ceph-mon[29254]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-20T07:58:04.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:04 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1187846617' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:58:04.325 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:04 smithi186 ceph-mon[36772]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-20T07:58:04.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.458 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-02-20T07:58:04.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.514 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list-id 2024-02-20T07:58:04.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.569 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2024-02-20T07:58:04.569 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-02-20T07:58:04.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-20T07:58:04.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-20T07:58:04.569 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-02-20T07:58:04.651 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-20T07:58:04.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.651 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2024-02-20T07:58:04.651 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-02-20T07:58:04.651 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-20T07:58:04.651 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-20T07:58:04.651 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-20T07:58:04.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-20T07:58:04.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-20T07:58:04.652 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-02-20T07:58:04.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.790 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2024-02-20T07:58:04.790 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-02-20T07:58:04.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.0 up 2024-02-20T07:58:04.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set dev brx.0 master ceph-brx 2024-02-20T07:58:04.791 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-02-20T07:58:04.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:58:04.831 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:04.886 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-20T07:58:04.886 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:58:04.887 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:04.941 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-20T07:58:04.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:04.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:58:05.047 INFO:teuthology.orchestra.run.smithi055.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:58:05.047 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-20T07:58:05.047 INFO:teuthology.orchestra.run.smithi055.stdout:invalid new device string format 2024-02-20T07:58:05.168 INFO:teuthology.orchestra.run.smithi055.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:58:05.168 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-20T07:58:05.168 INFO:teuthology.orchestra.run.smithi055.stdout:invalid new device string format 2024-02-20T07:58:05.168 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.186:3300" 2024-02-20T07:58:05.168 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:58:05.172 INFO:teuthology.orchestra.run.smithi055.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:58:05.172 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-20T07:58:05.172 INFO:teuthology.orchestra.run.smithi055.stdout:invalid new device string format 2024-02-20T07:58:05.172 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.186:3300" 2024-02-20T07:58:05.172 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: resolved to: "172.21.15.55:6789,172.21.15.186:6789" 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.55:6789,172.21.15.186:6789:/ 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=9a1162a4-cfc4-11ee-95bb-87774f69a715" will pass to kernel 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout: You just mounted a file system that supports labels which does not 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout: applications will generate AVC messages and not be allowed access to 2024-02-20T07:58:05.173 INFO:teuthology.orchestra.run.smithi055.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-20T07:58:05.175 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-20T07:58:05.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:05.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:05.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:05.206 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:58:05.265 INFO:teuthology.orchestra.run.smithi055.stdout:client24301 2024-02-20T07:58:05.266 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:58:05.266 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T07:58:05.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:05.322 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:58:05.382 INFO:teuthology.orchestra.run.smithi055.stdout:client24301 2024-02-20T07:58:05.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:05.383 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:58:05.443 INFO:teuthology.orchestra.run.smithi055.stdout:client24301 2024-02-20T07:58:05.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:05.443 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715.client24301/status') 2024-02-20T07:58:05.512 INFO:teuthology.orchestra.run.smithi055.stdout:instance: client.24301 (3)172.21.15.55:0/4018398591 2024-02-20T07:58:05.513 INFO:teuthology.orchestra.run.smithi055.stdout:blocklisted: false 2024-02-20T07:58:05.513 INFO:teuthology.orchestra.run.smithi055.stderr:0+1 records in 2024-02-20T07:58:05.513 INFO:teuthology.orchestra.run.smithi055.stderr:0+1 records out 2024-02-20T07:58:05.513 INFO:teuthology.orchestra.run.smithi055.stderr:71 bytes copied, 2.3129e-05 s, 3.1 MB/s 2024-02-20T07:58:05.514 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-20T07:58:05.514 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-20T07:58:05.514 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:05.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-20T07:58:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:06.464 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:06 smithi055 ceph-mon[29254]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-20T07:58:06.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:06 smithi186 ceph-mon[36772]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-20T07:58:07.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 -- 172.21.15.55:0/2168624540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 msgr2=0x7f147c103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:07.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 --2- 172.21.15.55:0/2168624540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c103510 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f146c004940 tx=0x7f146c009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:07.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 -- 172.21.15.55:0/2168624540 shutdown_connections 2024-02-20T07:58:07.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 --2- 172.21.15.55:0/2168624540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 0x7f147c0fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:07.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 --2- 172.21.15.55:0/2168624540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c103510 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:07.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 -- 172.21.15.55:0/2168624540 >> 172.21.15.55:0/2168624540 conn(0x7f147c071560 msgr2=0x7f147c071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:07.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 -- 172.21.15.55:0/2168624540 shutdown_connections 2024-02-20T07:58:07.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.233+0000 7f1483eff700 1 -- 172.21.15.55:0/2168624540 wait complete. 2024-02-20T07:58:07.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1483eff700 1 Processor -- start 2024-02-20T07:58:07.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1483eff700 1 -- start start 2024-02-20T07:58:07.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1483eff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 0x7f147c0fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:07.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1483eff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c0fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:07.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1483eff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0fd320 con 0x7f147c103140 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1483eff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0fd490 con 0x7f147c0fb570 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f148149a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c0fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f148149a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c0fcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43156/0 (socket says 172.21.15.55:43156) 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f148149a700 1 -- 172.21.15.55:0/2099204930 learned_addr learned my addr 172.21.15.55:0/2099204930 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1481c9b700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 0x7f147c0fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f148149a700 1 -- 172.21.15.55:0/2099204930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 msgr2=0x7f147c0fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:07.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f148149a700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 0x7f147c0fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:07.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f148149a700 1 -- 172.21.15.55:0/2099204930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c005040 con 0x7f147c103140 2024-02-20T07:58:07.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.234+0000 7f1481c9b700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 0x7f147c0fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:58:07.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.235+0000 7f148149a700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c0fcc90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f147800b790 tx=0x7f1478009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:07.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.235+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147800f770 con 0x7f147c103140 2024-02-20T07:58:07.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.235+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f147800fda0 con 0x7f147c103140 2024-02-20T07:58:07.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.235+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c119e90 con 0x7f147c103140 2024-02-20T07:58:07.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.235+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147800d310 con 0x7f147c103140 2024-02-20T07:58:07.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.235+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c11a350 con 0x7f147c103140 2024-02-20T07:58:07.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.236+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c073080 con 0x7f147c103140 2024-02-20T07:58:07.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.236+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1478010420 con 0x7f147c103140 2024-02-20T07:58:07.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.237+0000 7f14727fc700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f14680692a0 0x7f146806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:07.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.237+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f14780872c0 con 0x7f147c103140 2024-02-20T07:58:07.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.237+0000 7f1481c9b700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f14680692a0 0x7f146806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:07.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.237+0000 7f1481c9b700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f14680692a0 0x7f146806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f146c004560 tx=0x7f146c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:07.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.241+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1478053500 con 0x7f147c103140 2024-02-20T07:58:07.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.478+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f147c0fde90 con 0x7f147c103140 2024-02-20T07:58:07.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.478+0000 7f14727fc700 1 -- 172.21.15.55:0/2099204930 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f1478056b20 con 0x7f147c103140 2024-02-20T07:58:07.479 INFO:teuthology.orchestra.run.smithi055.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-20T07:58:07.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f14680692a0 msgr2=0x7f146806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f14680692a0 0x7f146806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f146c004560 tx=0x7f146c009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 msgr2=0x7f147c0fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c0fcc90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f147800b790 tx=0x7f1478009250 comp rx=0 tx=0).stop 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 shutdown_connections 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f147c0fb570 0x7f147c0fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f14680692a0 0x7f146806b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 --2- 172.21.15.55:0/2099204930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f147c103140 0x7f147c0fcc90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.481+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 >> 172.21.15.55:0/2099204930 conn(0x7f147c071560 msgr2=0x7f147c0f96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.482+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 shutdown_connections 2024-02-20T07:58:07.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:07.482+0000 7f1483eff700 1 -- 172.21.15.55:0/2099204930 wait complete. 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi186.front.sepia.ceph.com 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-20T07:58:08.021 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-20T07:58:08.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.022 DEBUG:teuthology.orchestra.run.smithi186:> ip addr 2024-02-20T07:58:08.042 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-20T07:58:08.042 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-20T07:58:08.042 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:58:08.043 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-20T07:58:08.044 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42241sec preferred_lft 42241sec 2024-02-20T07:58:08.044 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::ec4:7aff:fe88:6df5/64 scope link 2024-02-20T07:58:08.044 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:58:08.044 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-20T07:58:08.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.044 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-02-20T07:58:08.044 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-02-20T07:58:08.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add name ceph-brx type bridge 2024-02-20T07:58:08.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr flush dev ceph-brx 2024-02-20T07:58:08.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set ceph-brx up 2024-02-20T07:58:08.045 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-20T07:58:08.045 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-02-20T07:58:08.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.217 DEBUG:teuthology.orchestra.run.smithi186:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-20T07:58:08.243 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-02-20T07:58:08.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.245 DEBUG:teuthology.orchestra.run.smithi186:> route 2024-02-20T07:58:08.312 INFO:teuthology.orchestra.run.smithi186.stdout:Kernel IP routing table 2024-02-20T07:58:08.312 INFO:teuthology.orchestra.run.smithi186.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-20T07:58:08.312 INFO:teuthology.orchestra.run.smithi186.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-20T07:58:08.312 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-20T07:58:08.312 INFO:teuthology.orchestra.run.smithi186.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-20T07:58:08.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.313 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-02-20T07:58:08.313 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-02-20T07:58:08.313 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-20T07:58:08.313 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-20T07:58:08.313 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-20T07:58:08.313 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-02-20T07:58:08.378 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:08 smithi186 ceph-mon[36772]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:58:08.378 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:08 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/2099204930' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:58:08.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.570 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-02-20T07:58:08.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.626 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list-id 2024-02-20T07:58:08.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.681 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-02-20T07:58:08.681 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-02-20T07:58:08.681 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-20T07:58:08.682 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-20T07:58:08.682 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-02-20T07:58:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:08 smithi055 ceph-mon[29254]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:58:08.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:08 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/2099204930' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:58:08.763 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-20T07:58:08.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.763 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-02-20T07:58:08.763 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-02-20T07:58:08.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-20T07:58:08.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-20T07:58:08.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-20T07:58:08.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-20T07:58:08.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-20T07:58:08.764 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-02-20T07:58:08.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:08.901 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-02-20T07:58:08.901 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-02-20T07:58:08.901 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set brx.0 up 2024-02-20T07:58:08.901 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set dev brx.0 master ceph-brx 2024-02-20T07:58:08.901 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-02-20T07:58:08.942 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:58:08.942 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:08.997 INFO:teuthology.orchestra.run.smithi186.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-20T07:58:08.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:58:08.998 DEBUG:teuthology.orchestra.run.smithi186:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:09.052 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-20T07:58:09.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:09.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:58:09.141 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:58:09.141 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-20T07:58:09.141 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-02-20T07:58:09.261 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:58:09.261 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-20T07:58:09.261 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-02-20T07:58:09.261 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.186:3300" 2024-02-20T07:58:09.261 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.186:3300" 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.55:6789,172.21.15.186:6789" 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.55:6789,172.21.15.186:6789:/ 2024-02-20T07:58:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=9a1162a4-cfc4-11ee-95bb-87774f69a715" will pass to kernel 2024-02-20T07:58:09.266 INFO:teuthology.orchestra.run.smithi186.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-20T07:58:09.266 INFO:teuthology.orchestra.run.smithi186.stdout: You just mounted a file system that supports labels which does not 2024-02-20T07:58:09.266 INFO:teuthology.orchestra.run.smithi186.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-20T07:58:09.266 INFO:teuthology.orchestra.run.smithi186.stdout: applications will generate AVC messages and not be allowed access to 2024-02-20T07:58:09.266 INFO:teuthology.orchestra.run.smithi186.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-20T07:58:09.268 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-20T07:58:09.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:09.268 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:09.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:09.295 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:58:09.352 INFO:teuthology.orchestra.run.smithi186.stdout:client24303 2024-02-20T07:58:09.352 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:58:09.353 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T07:58:09.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:09.409 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:58:09.464 INFO:teuthology.orchestra.run.smithi186.stdout:client24303 2024-02-20T07:58:09.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:09.465 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:58:09.520 INFO:teuthology.orchestra.run.smithi186.stdout:client24303 2024-02-20T07:58:09.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:58:09.521 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715.client24303/status') 2024-02-20T07:58:09.587 INFO:teuthology.orchestra.run.smithi186.stdout:instance: client.24303 (3)192.168.0.1:0/2261739733 2024-02-20T07:58:09.587 INFO:teuthology.orchestra.run.smithi186.stdout:blocklisted: false 2024-02-20T07:58:09.587 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records in 2024-02-20T07:58:09.588 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records out 2024-02-20T07:58:09.588 INFO:teuthology.orchestra.run.smithi186.stderr:70 bytes copied, 2.137e-05 s, 3.3 MB/s 2024-02-20T07:58:09.589 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:58:09.599 INFO:teuthology.task.print:**** done client 2024-02-20T07:58:09.599 INFO:teuthology.run_tasks:Running task parallel... 2024-02-20T07:58:09.615 INFO:teuthology.task.parallel:starting parallel... 2024-02-20T07:58:09.615 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-20T07:58:09.616 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-20T07:58:09.616 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T07:58:09.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-20T07:58:09.617 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-20T07:58:09.617 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-20T07:58:09.620 INFO:tasks.workunit:Pulling workunits from ref 3d72bcfd7affbb4a923c191d212814b4934cfeaa 2024-02-20T07:58:09.621 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-20T07:58:09.621 INFO:tasks.workunit:timeout=3h 2024-02-20T07:58:09.621 INFO:tasks.workunit:cleanup=True 2024-02-20T07:58:09.621 DEBUG:teuthology.orchestra.run.smithi055:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 46h/70d Inode: 1 Links: 2 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-02-20 07:57:35.061844483 +0000 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-02-20 07:58:05.199215485 +0000 2024-02-20T07:58:09.641 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-02-20 07:57:35.061844483 +0000 2024-02-20T07:58:09.642 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-20T07:58:09.642 DEBUG:teuthology.orchestra.run.smithi055:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-20T07:58:09.716 DEBUG:teuthology.orchestra.run.smithi186:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:09.734 INFO:teuthology.orchestra.run.smithi186.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:09.734 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-20T07:58:09.734 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-20T07:58:09.734 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:58:09.734 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-20T07:58:09.734 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-20T07:58:09.735 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-02-20 07:58:09.708125786 +0000 2024-02-20T07:58:09.735 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-02-20 07:58:09.708125786 +0000 2024-02-20T07:58:09.735 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-02-20 07:57:35.061844483 +0000 2024-02-20T07:58:09.735 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-20T07:58:09.735 DEBUG:teuthology.orchestra.run.smithi186:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-20T07:58:09.809 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d72bcfd7affbb4a923c191d212814b4934cfeaa 2024-02-20T07:58:09.810 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 3d72bcfd7affbb4a923c191d212814b4934cfeaa 2024-02-20T07:58:09.825 INFO:tasks.workunit.client.0.smithi055.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-20T07:58:09.865 INFO:tasks.workunit.client.1.smithi186.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-20T07:58:09.902 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:10.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:10 smithi055 ceph-mon[29254]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-20T07:58:10.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:10 smithi186 ceph-mon[36772]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-20T07:58:11.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.492+0000 7f09444e1700 1 -- 172.21.15.55:0/2830187824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fe0e0 msgr2=0x7f093c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:11.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.492+0000 7f09444e1700 1 --2- 172.21.15.55:0/2830187824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fe0e0 0x7f093c0fe550 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f092c004940 tx=0x7f092c009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- 172.21.15.55:0/2830187824 shutdown_connections 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 --2- 172.21.15.55:0/2830187824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fe0e0 0x7f093c0fe550 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 --2- 172.21.15.55:0/2830187824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c0fcef0 0x7f093c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- 172.21.15.55:0/2830187824 >> 172.21.15.55:0/2830187824 conn(0x7f093c0f8620 msgr2=0x7f093c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- 172.21.15.55:0/2830187824 shutdown_connections 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- 172.21.15.55:0/2830187824 wait complete. 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 Processor -- start 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- start start 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 0x7f093c10d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c0fe0e0 0x7f093c10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c10e4b0 con 0x7f093c0fcef0 2024-02-20T07:58:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.493+0000 7f09444e1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c10ed50 con 0x7f093c0fe0e0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 0x7f093c10d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 0x7f093c10d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43164/0 (socket says 172.21.15.55:43164) 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 -- 172.21.15.55:0/2393948961 learned_addr learned my addr 172.21.15.55:0/2393948961 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f0941a7c700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c0fe0e0 0x7f093c10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 -- 172.21.15.55:0/2393948961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c0fe0e0 msgr2=0x7f093c10deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c0fe0e0 0x7f093c10deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 -- 172.21.15.55:0/2393948961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c005040 con 0x7f093c0fcef0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f094227d700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 0x7f093c10d970 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0938002a80 tx=0x7f0938004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0938007cc0 con 0x7f093c0fcef0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f093800f040 con 0x7f093c0fcef0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c10f030 con 0x7f093c0fcef0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f093800da30 con 0x7f093c0fcef0 2024-02-20T07:58:11.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.494+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c10f570 con 0x7f093c0fcef0 2024-02-20T07:58:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.496+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c04bed0 con 0x7f093c0fcef0 2024-02-20T07:58:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.496+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f093800db90 con 0x7f093c0fcef0 2024-02-20T07:58:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.497+0000 7f0932ffd700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f09280692a0 0x7f092806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.497+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0938086e90 con 0x7f093c0fcef0 2024-02-20T07:58:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.497+0000 7f0941a7c700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f09280692a0 0x7f092806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:11.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.497+0000 7f0941a7c700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f09280692a0 0x7f092806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f092c00b3b0 tx=0x7f092c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.501+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0938056c00 con 0x7f093c0fcef0 2024-02-20T07:58:11.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.720+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f093c10f1c0 con 0x7f093c0fcef0 2024-02-20T07:58:11.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.720+0000 7f0932ffd700 1 -- 172.21.15.55:0/2393948961 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f093c10f1c0 con 0x7f093c0fcef0 2024-02-20T07:58:11.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.722+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f09280692a0 msgr2=0x7f092806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:11.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.722+0000 7f09444e1700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f09280692a0 0x7f092806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f092c00b3b0 tx=0x7f092c009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.722+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 msgr2=0x7f093c10d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.722+0000 7f09444e1700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 0x7f093c10d970 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0938002a80 tx=0x7f0938004600 comp rx=0 tx=0).stop 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 shutdown_connections 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f09280692a0 0x7f092806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f093c0fcef0 0x7f093c10d970 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 --2- 172.21.15.55:0/2393948961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c0fe0e0 0x7f093c10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 >> 172.21.15.55:0/2393948961 conn(0x7f093c0f8620 msgr2=0x7f093c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 shutdown_connections 2024-02-20T07:58:11.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:11.723+0000 7f09444e1700 1 -- 172.21.15.55:0/2393948961 wait complete. 2024-02-20T07:58:12.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-20T07:58:12.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:12 smithi186 ceph-mon[36772]: pgmap v198: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:12.674 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:12.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:12 smithi055 ceph-mon[29254]: pgmap v198: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:14.185 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:14 smithi055 ceph-mon[29254]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-02-20T07:58:14.185 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:14 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:14.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:14 smithi186 ceph-mon[36772]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-02-20T07:58:14.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:14 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 -- 172.21.15.55:0/2887306115 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fbe50 msgr2=0x7f79f80fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 --2- 172.21.15.55:0/2887306115 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fbe50 0x7f79f80fc260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f79e8004940 tx=0x7f79e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 -- 172.21.15.55:0/2887306115 shutdown_connections 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 --2- 172.21.15.55:0/2887306115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f80fcfc0 0x7f79f80fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 --2- 172.21.15.55:0/2887306115 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fbe50 0x7f79f80fc260 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 -- 172.21.15.55:0/2887306115 >> 172.21.15.55:0/2887306115 conn(0x7f79f80f7510 msgr2=0x7f79f80f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 -- 172.21.15.55:0/2887306115 shutdown_connections 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 -- 172.21.15.55:0/2887306115 wait complete. 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 Processor -- start 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.416+0000 7f79fd80e700 1 -- start start 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79fd80e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f80fbe50 0x7f79f810fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:14.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79fd80e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 0x7f79f810fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:14.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79fd80e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f81105f0 con 0x7f79f80fcfc0 2024-02-20T07:58:14.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79fd80e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8110e90 con 0x7f79f80fbe50 2024-02-20T07:58:14.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 0x7f79f810fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:14.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 0x7f79f810fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50150/0 (socket says 172.21.15.55:50150) 2024-02-20T07:58:14.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 -- 172.21.15.55:0/4222349770 learned_addr learned my addr 172.21.15.55:0/4222349770 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:14.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f6ffd700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f80fbe50 0x7f79f810fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 -- 172.21.15.55:0/4222349770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f80fbe50 msgr2=0x7f79f810fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f80fbe50 0x7f79f810fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 -- 172.21.15.55:0/4222349770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8005040 con 0x7f79f80fcfc0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79f67fc700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 0x7f79f810fff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00eb40 tx=0x7f79ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.417+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79ec007cc0 con 0x7f79f80fcfc0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.418+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79ec00f040 con 0x7f79f80fcfc0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.418+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f8111110 con 0x7f79f80fcfc0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.418+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79ec00ca10 con 0x7f79f80fcfc0 2024-02-20T07:58:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.418+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f8111650 con 0x7f79f80fcfc0 2024-02-20T07:58:14.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.419+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f79ec00cc80 con 0x7f79f80fcfc0 2024-02-20T07:58:14.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.420+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f804bed0 con 0x7f79f80fcfc0 2024-02-20T07:58:14.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.420+0000 7f79df7fe700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f79e00692a0 0x7f79e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:14.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.420+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f79ec086dd0 con 0x7f79f80fcfc0 2024-02-20T07:58:14.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.420+0000 7f79f6ffd700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f79e00692a0 0x7f79e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:14.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.420+0000 7f79f6ffd700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f79e00692a0 0x7f79e006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009b30 tx=0x7f79e8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:14.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.425+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f79ec04dbd0 con 0x7f79f80fcfc0 2024-02-20T07:58:14.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.621+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f79f8104940 con 0x7f79f80fcfc0 2024-02-20T07:58:14.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.621+0000 7f79df7fe700 1 -- 172.21.15.55:0/4222349770 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f79f8104940 con 0x7f79f80fcfc0 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.623+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f79e00692a0 msgr2=0x7f79e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.623+0000 7f79fd80e700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f79e00692a0 0x7f79e006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009b30 tx=0x7f79e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.623+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 msgr2=0x7f79f810fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.623+0000 7f79fd80e700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 0x7f79f810fff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00eb40 tx=0x7f79ec00b040 comp rx=0 tx=0).stop 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.623+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 shutdown_connections 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.624+0000 7f79fd80e700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79f80fbe50 0x7f79f810fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.624+0000 7f79fd80e700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f79e00692a0 0x7f79e006b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.624+0000 7f79fd80e700 1 --2- 172.21.15.55:0/4222349770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79f80fcfc0 0x7f79f810fff0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.624+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 >> 172.21.15.55:0/4222349770 conn(0x7f79f80f7510 msgr2=0x7f79f8065f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.624+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 shutdown_connections 2024-02-20T07:58:14.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:14.624+0000 7f79fd80e700 1 -- 172.21.15.55:0/4222349770 wait complete. 2024-02-20T07:58:15.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-20T07:58:15.685 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:16.718 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:16 smithi055 ceph-mon[29254]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-02-20T07:58:16.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:16 smithi186 ceph-mon[36772]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-02-20T07:58:17.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 -- 172.21.15.55:0/2632257444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fbe40 msgr2=0x7fcfd40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:17.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 --2- 172.21.15.55:0/2632257444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fbe40 0x7fcfd40fc250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4004940 tx=0x7fcfc4009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:17.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 -- 172.21.15.55:0/2632257444 shutdown_connections 2024-02-20T07:58:17.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 --2- 172.21.15.55:0/2632257444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:17.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 --2- 172.21.15.55:0/2632257444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fbe40 0x7fcfd40fc250 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:17.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 -- 172.21.15.55:0/2632257444 >> 172.21.15.55:0/2632257444 conn(0x7fcfd40f7520 msgr2=0x7fcfd40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:17.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 -- 172.21.15.55:0/2632257444 shutdown_connections 2024-02-20T07:58:17.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 -- 172.21.15.55:0/2632257444 wait complete. 2024-02-20T07:58:17.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 Processor -- start 2024-02-20T07:58:17.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.454+0000 7fcfda022700 1 -- start start 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfda022700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd40fbe40 0x7fcfd410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfda022700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfda022700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd410e470 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfda022700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd410ed10 con 0x7fcfd40fbe40 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd40fbe40 0x7fcfd410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50174/0 (socket says 172.21.15.55:50174) 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 -- 172.21.15.55:0/14239467 learned_addr learned my addr 172.21.15.55:0/14239467 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:17.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 -- 172.21.15.55:0/14239467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd40fbe40 msgr2=0x7fcfd410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd40fbe40 0x7fcfd410d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 -- 172.21.15.55:0/14239467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4005040 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.455+0000 7fcfd2ffd700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd410de70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800c830 tx=0x7fcfc8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.456+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfc80106f0 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.456+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfc8010d20 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.456+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd410eff0 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.456+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfc80113a0 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.456+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd410f530 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.457+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd404bed0 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.457+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcfc8012400 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.458+0000 7fcfbbfff700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcfbc0692b0 0x7fcfbc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:17.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.458+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcfc80870b0 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.458+0000 7fcfd37fe700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcfbc0692b0 0x7fcfbc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:17.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.458+0000 7fcfd37fe700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcfbc0692b0 0x7fcfbc06b760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400b7a0 tx=0x7fcfc4009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:17.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.462+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcfc8056ec0 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.671+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fcfd4104930 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.672+0000 7fcfbbfff700 1 -- 172.21.15.55:0/14239467 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fcfc8056a50 con 0x7fcfd40fcfb0 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcfbc0692b0 msgr2=0x7fcfbc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcfbc0692b0 0x7fcfbc06b760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400b7a0 tx=0x7fcfc4009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 msgr2=0x7fcfd410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd410de70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800c830 tx=0x7fcfc8004470 comp rx=0 tx=0).stop 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 shutdown_connections 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd40fbe40 0x7fcfd410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fcfbc0692b0 0x7fcfbc06b760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 --2- 172.21.15.55:0/14239467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd40fcfb0 0x7fcfd410de70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 >> 172.21.15.55:0/14239467 conn(0x7fcfd40f7520 msgr2=0x7fcfd4065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:17.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 shutdown_connections 2024-02-20T07:58:17.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:17.674+0000 7fcfda022700 1 -- 172.21.15.55:0/14239467 wait complete. 2024-02-20T07:58:18.422 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:18 smithi055 ceph-mon[29254]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-02-20T07:58:18.449 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-20T07:58:18.576 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:18 smithi186 ceph-mon[36772]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-02-20T07:58:18.729 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:20.542 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:20 smithi055 ceph-mon[29254]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 -- 172.21.15.55:0/823095751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fcfb0 msgr2=0x7f647c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 --2- 172.21.15.55:0/823095751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fcfb0 0x7f647c0fd420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f646c009ce0 tx=0x7f646c004d10 comp rx=0 tx=0).stop 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 -- 172.21.15.55:0/823095751 shutdown_connections 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 --2- 172.21.15.55:0/823095751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fcfb0 0x7f647c0fd420 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 --2- 172.21.15.55:0/823095751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f647c0fbe40 0x7f647c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 -- 172.21.15.55:0/823095751 >> 172.21.15.55:0/823095751 conn(0x7f647c0f74e0 msgr2=0x7f647c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 -- 172.21.15.55:0/823095751 shutdown_connections 2024-02-20T07:58:20.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.544+0000 7f64815d5700 1 -- 172.21.15.55:0/823095751 wait complete. 2024-02-20T07:58:20.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f64815d5700 1 Processor -- start 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f64815d5700 1 -- start start 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f64815d5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 0x7f647c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f64815d5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f647c0fcfb0 0x7f647c10df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f64815d5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c10e540 con 0x7f647c0fbe40 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f64815d5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c10ede0 con 0x7f647c0fcfb0 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 0x7f647c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 0x7f647c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50196/0 (socket says 172.21.15.55:50196) 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 -- 172.21.15.55:0/3431863371 learned_addr learned my addr 172.21.15.55:0/3431863371 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647a7fc700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f647c0fcfb0 0x7f647c10df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 -- 172.21.15.55:0/3431863371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f647c0fcfb0 msgr2=0x7f647c10df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f647c0fcfb0 0x7f647c10df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 -- 172.21.15.55:0/3431863371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009940 con 0x7f647c0fbe40 2024-02-20T07:58:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.545+0000 7f647affd700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 0x7f647c10da00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f647400eb70 tx=0x7f647400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:20.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.546+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6474007cc0 con 0x7f647c0fbe40 2024-02-20T07:58:20.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.546+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f647400f040 con 0x7f647c0fbe40 2024-02-20T07:58:20.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.546+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c10f060 con 0x7f647c0fbe40 2024-02-20T07:58:20.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.546+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647400ca90 con 0x7f647c0fbe40 2024-02-20T07:58:20.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.546+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c10f5a0 con 0x7f647c0fbe40 2024-02-20T07:58:20.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.547+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c04bed0 con 0x7f647c0fbe40 2024-02-20T07:58:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.547+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f647400cbf0 con 0x7f647c0fbe40 2024-02-20T07:58:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.548+0000 7f64737fe700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64680692b0 0x7f646806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.548+0000 7f647a7fc700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64680692b0 0x7f646806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.548+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6474086cd0 con 0x7f647c0fbe40 2024-02-20T07:58:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.548+0000 7f647a7fc700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64680692b0 0x7f646806b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f646c015040 tx=0x7f646c004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.552+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6474056a40 con 0x7f647c0fbe40 2024-02-20T07:58:20.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:20.762+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f647c10fa00 con 0x7f64680692b0 2024-02-20T07:58:20.829 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:20 smithi186 ceph-mon[36772]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-02-20T07:58:22.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:22 smithi186 ceph-mon[36772]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:22.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:22 smithi186 ceph-mon[36772]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-20T07:58:22.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:22 smithi055 ceph-mon[29254]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:22.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:22 smithi055 ceph-mon[29254]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-20T07:58:24.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:24 smithi055 ceph-mon[29254]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-20T07:58:24.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:24 smithi186 ceph-mon[36772]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-20T07:58:26.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:26 smithi055 ceph-mon[29254]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:58:26.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:26 smithi186 ceph-mon[36772]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:58:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:28 smithi055 ceph-mon[29254]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:58:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:28 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:28.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:28 smithi186 ceph-mon[36772]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:58:28.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:28 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:30.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:30 smithi055 ceph-mon[29254]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:30.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:30 smithi186 ceph-mon[36772]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:32.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:32 smithi186 ceph-mon[36772]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:32.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:32 smithi055 ceph-mon[29254]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:34.324 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:34 smithi186 ceph-mon[36772]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:34.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:34 smithi055 ceph-mon[29254]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:36.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:36 smithi055 ceph-mon[29254]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:36.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:36 smithi186 ceph-mon[36772]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:38 smithi055 ceph-mon[29254]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:38.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:38 smithi186 ceph-mon[36772]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:40.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:40 smithi055 ceph-mon[29254]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:40.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:40 smithi186 ceph-mon[36772]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:42.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:42 smithi186 ceph-mon[36772]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:42.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:42 smithi055 ceph-mon[29254]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:44.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:44 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:44.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:44 smithi055 ceph-mon[29254]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:44.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:44 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:44.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:44 smithi186 ceph-mon[36772]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:46.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:46 smithi055 ceph-mon[29254]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:46.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:46 smithi186 ceph-mon[36772]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:48 smithi055 ceph-mon[29254]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:48.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:48 smithi186 ceph-mon[36772]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:58:49.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.223+0000 7f64737fe700 1 -- 172.21.15.55:0/3431863371 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f647c10fa00 con 0x7f64680692b0 2024-02-20T07:58:49.225 INFO:teuthology.orchestra.run.smithi055.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64680692b0 msgr2=0x7f646806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64680692b0 0x7f646806b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f646c015040 tx=0x7f646c004d10 comp rx=0 tx=0).stop 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 msgr2=0x7f647c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 0x7f647c10da00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f647400eb70 tx=0x7f647400b040 comp rx=0 tx=0).stop 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 shutdown_connections 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f64680692b0 0x7f646806b760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c0fbe40 0x7f647c10da00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:49.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 --2- 172.21.15.55:0/3431863371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f647c0fcfb0 0x7f647c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:49.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.226+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 >> 172.21.15.55:0/3431863371 conn(0x7f647c0f74e0 msgr2=0x7f647c0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:49.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.227+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 shutdown_connections 2024-02-20T07:58:49.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:49.227+0000 7f64815d5700 1 -- 172.21.15.55:0/3431863371 wait complete. 2024-02-20T07:58:50.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-20T07:58:50.446 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T07:58:50.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:58:50.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:50.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:58:50.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-20T07:58:50.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:58:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:58:50.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:50 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:50.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:58:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:58:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-20T07:58:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:58:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:58:50.575 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:50 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:51.384 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:51 smithi055 ceph-mon[29254]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:58:51.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:51 smithi186 ceph-mon[36772]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:58:52.432 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:52 smithi055 ceph-mon[29254]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:52.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:52 smithi186 ceph-mon[36772]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:52.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 -- 172.21.15.55:0/3699059159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c240fcfb0 msgr2=0x7f7c240fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:52.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3699059159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c240fcfb0 0x7f7c240fd420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7c14004940 tx=0x7f7c14009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:52.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 -- 172.21.15.55:0/3699059159 shutdown_connections 2024-02-20T07:58:52.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3699059159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c240fcfb0 0x7f7c240fd420 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7c14004940 tx=0x7f7c14009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:52.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3699059159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c240fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 -- 172.21.15.55:0/3699059159 >> 172.21.15.55:0/3699059159 conn(0x7f7c240f7500 msgr2=0x7f7c240f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 -- 172.21.15.55:0/3699059159 shutdown_connections 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.872+0000 7f7c2c385700 1 -- 172.21.15.55:0/3699059159 wait complete. 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.873+0000 7f7c2c385700 1 Processor -- start 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.873+0000 7f7c2c385700 1 -- start start 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.873+0000 7f7c2c385700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c2410b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:52.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.873+0000 7f7c2c385700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 0x7f7c2410cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:52.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c29920700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 0x7f7c2410cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:52.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c29920700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 0x7f7c2410cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48788/0 (socket says 172.21.15.55:48788) 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c2a121700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c2410b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c2a121700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c2410b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.55:49692/0 (socket says 172.21.15.55:49692) 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c29920700 1 -- 172.21.15.55:0/3408958018 learned_addr learned my addr 172.21.15.55:0/3408958018 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2410c210 con 0x7f7c2410bd50 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2410c380 con 0x7f7c240fbe40 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c29920700 1 -- 172.21.15.55:0/3408958018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 msgr2=0x7f7c2410b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.874+0000 7f7c29920700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c2410b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c29920700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c14005040 con 0x7f7c2410bd50 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c2a121700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c2410b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c29920700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 0x7f7c2410cc40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400b3b0 tx=0x7f7c14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1400e040 con 0x7f7c2410bd50 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2410d240 con 0x7f7c2410bd50 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c14004290 con 0x7f7c2410bd50 2024-02-20T07:58:52.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.875+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1401c610 con 0x7f7c2410bd50 2024-02-20T07:58:52.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.876+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2410d730 con 0x7f7c2410bd50 2024-02-20T07:58:52.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.877+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2404bed0 con 0x7f7c2410bd50 2024-02-20T07:58:52.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.878+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7c14013a80 con 0x7f7c2410bd50 2024-02-20T07:58:52.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.878+0000 7f7c1affd700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7c100692b0 0x7f7c1006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:52.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.878+0000 7f7c2a121700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7c100692b0 0x7f7c1006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:52.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.879+0000 7f7c2a121700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7c100692b0 0x7f7c1006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009b80 tx=0x7f7c2000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:52.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.887+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7c1400be20 con 0x7f7c2410bd50 2024-02-20T07:58:52.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:52.887+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7c1408d060 con 0x7f7c2410bd50 2024-02-20T07:58:53.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.118+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c24102b80 con 0x7f7c100692b0 2024-02-20T07:58:53.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.119+0000 7f7c1affd700 1 -- 172.21.15.55:0/3408958018 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f7c24102b80 con 0x7f7c100692b0 2024-02-20T07:58:53.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.121+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7c100692b0 msgr2=0x7f7c1006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.121+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7c100692b0 0x7f7c1006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009b80 tx=0x7f7c2000b040 comp rx=0 tx=0).stop 2024-02-20T07:58:53.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.121+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 msgr2=0x7f7c2410cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.121+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 0x7f7c2410cc40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400b3b0 tx=0x7f7c14009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 shutdown_connections 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c240fbe40 0x7f7c2410b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f7c100692b0 0x7f7c1006b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 --2- 172.21.15.55:0/3408958018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2410bd50 0x7f7c2410cc40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 >> 172.21.15.55:0/3408958018 conn(0x7f7c240f7500 msgr2=0x7f7c24100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 shutdown_connections 2024-02-20T07:58:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.122+0000 7f7c2c385700 1 -- 172.21.15.55:0/3408958018 wait complete. 2024-02-20T07:58:53.136 INFO:teuthology.orchestra.run.smithi055.stdout:true 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.233+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/1681681217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d441003a0 msgr2=0x7f0d44102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.233+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/1681681217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d441003a0 0x7f0d44102780 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0d34004940 tx=0x7f0d34009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.233+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/1681681217 shutdown_connections 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.233+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/1681681217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d441003a0 0x7f0d44102780 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.233+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/1681681217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d440fda80 0x7f0d440ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.233+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/1681681217 >> 172.21.15.55:0/1681681217 conn(0x7f0d440f7500 msgr2=0x7f0d440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.234+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/1681681217 shutdown_connections 2024-02-20T07:58:53.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.234+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/1681681217 wait complete. 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.234+0000 7f0d4c0c2700 1 Processor -- start 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.234+0000 7f0d4c0c2700 1 -- start start 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d4c0c2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 0x7f0d4410d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d4c0c2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441003a0 0x7f0d4410de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d4c0c2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4410e400 con 0x7f0d440fda80 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d4c0c2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4410eca0 con 0x7f0d441003a0 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 0x7f0d4410d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 0x7f0d4410d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48814/0 (socket says 172.21.15.55:48814) 2024-02-20T07:58:53.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 -- 172.21.15.55:0/3149372297 learned_addr learned my addr 172.21.15.55:0/3149372297 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d4965d700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441003a0 0x7f0d4410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 -- 172.21.15.55:0/3149372297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441003a0 msgr2=0x7f0d4410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441003a0 0x7f0d4410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 -- 172.21.15.55:0/3149372297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34005040 con 0x7f0d440fda80 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.235+0000 7f0d49e5e700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 0x7f0d4410d8c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0d40002a80 tx=0x7f0d400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.236+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d40007cc0 con 0x7f0d440fda80 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.236+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d4000f040 con 0x7f0d440fda80 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.236+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d4000dac0 con 0x7f0d440fda80 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.236+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4410ef80 con 0x7f0d440fda80 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.236+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4410f4c0 con 0x7f0d440fda80 2024-02-20T07:58:53.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.237+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d44107a50 con 0x7f0d440fda80 2024-02-20T07:58:53.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.237+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0d4000dc20 con 0x7f0d440fda80 2024-02-20T07:58:53.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.238+0000 7f0d3a7fc700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0d300692a0 0x7f0d3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:53.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.238+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0d40086d60 con 0x7f0d440fda80 2024-02-20T07:58:53.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.238+0000 7f0d4965d700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0d300692a0 0x7f0d3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:53.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.239+0000 7f0d4965d700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0d300692a0 0x7f0d3006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400b3b0 tx=0x7f0d34009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:53.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.243+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0d40056ad0 con 0x7f0d440fda80 2024-02-20T07:58:53.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.664+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d4410f7a0 con 0x7f0d300692a0 2024-02-20T07:58:53.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.666+0000 7f0d3a7fc700 1 -- 172.21.15.55:0/3149372297 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f0d4410f7a0 con 0x7f0d300692a0 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.668+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0d300692a0 msgr2=0x7f0d3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.668+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0d300692a0 0x7f0d3006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400b3b0 tx=0x7f0d34009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.668+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 msgr2=0x7f0d4410d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.668+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 0x7f0d4410d8c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0d40002a80 tx=0x7f0d400046b0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 shutdown_connections 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f0d300692a0 0x7f0d3006b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d440fda80 0x7f0d4410d8c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 --2- 172.21.15.55:0/3149372297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441003a0 0x7f0d4410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 >> 172.21.15.55:0/3149372297 conn(0x7f0d440f7500 msgr2=0x7f0d440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 shutdown_connections 2024-02-20T07:58:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.669+0000 7f0d4c0c2700 1 -- 172.21.15.55:0/3149372297 wait complete. 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 -- 172.21.15.55:0/1271314407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fcfb0 msgr2=0x7f5ae80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 --2- 172.21.15.55:0/1271314407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fcfb0 0x7f5ae80fd420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4004940 tx=0x7f5ad4009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 -- 172.21.15.55:0/1271314407 shutdown_connections 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 --2- 172.21.15.55:0/1271314407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fcfb0 0x7f5ae80fd420 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 --2- 172.21.15.55:0/1271314407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ae80fbe40 0x7f5ae80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 -- 172.21.15.55:0/1271314407 >> 172.21.15.55:0/1271314407 conn(0x7f5ae80f74e0 msgr2=0x7f5ae80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 -- 172.21.15.55:0/1271314407 shutdown_connections 2024-02-20T07:58:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 -- 172.21.15.55:0/1271314407 wait complete. 2024-02-20T07:58:53.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.774+0000 7f5aef17d700 1 Processor -- start 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aef17d700 1 -- start start 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aef17d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 0x7f5ae810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aef17d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ae80fcfb0 0x7f5ae810dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aef17d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae810e4a0 con 0x7f5ae80fbe40 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aef17d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae810ed40 con 0x7f5ae80fcfb0 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aecf19700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 0x7f5ae810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aecf19700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 0x7f5ae810d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48836/0 (socket says 172.21.15.55:48836) 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aecf19700 1 -- 172.21.15.55:0/3438576514 learned_addr learned my addr 172.21.15.55:0/3438576514 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5ae7fff700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ae80fcfb0 0x7f5ae810dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aecf19700 1 -- 172.21.15.55:0/3438576514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ae80fcfb0 msgr2=0x7f5ae810dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aecf19700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ae80fcfb0 0x7f5ae810dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.775+0000 7f5aecf19700 1 -- 172.21.15.55:0/3438576514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4005040 con 0x7f5ae80fbe40 2024-02-20T07:58:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.776+0000 7f5aecf19700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 0x7f5ae810d960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5adc002a50 tx=0x7f5adc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.776+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5adc007cc0 con 0x7f5ae80fbe40 2024-02-20T07:58:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.776+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5adc00f040 con 0x7f5ae80fbe40 2024-02-20T07:58:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.776+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae810f020 con 0x7f5ae80fbe40 2024-02-20T07:58:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.776+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5adc00da30 con 0x7f5ae80fbe40 2024-02-20T07:58:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.776+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae810f560 con 0x7f5ae80fbe40 2024-02-20T07:58:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.777+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae804bed0 con 0x7f5ae80fbe40 2024-02-20T07:58:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.778+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5adc00db90 con 0x7f5ae80fbe40 2024-02-20T07:58:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.778+0000 7f5ae57fa700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5ad8069360 0x7f5ad806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.779+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5adc086dd0 con 0x7f5ae80fbe40 2024-02-20T07:58:53.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.779+0000 7f5ae7fff700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5ad8069360 0x7f5ad806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:53.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.779+0000 7f5ae7fff700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5ad8069360 0x7f5ad806b810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40043d0 tx=0x7f5ad4009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:53.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:53.783+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5adc056be0 con 0x7f5ae80fbe40 2024-02-20T07:58:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.118+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ae810f9d0 con 0x7f5ad8069360 2024-02-20T07:58:54.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.136+0000 7f5ae57fa700 1 -- 172.21.15.55:0/3438576514 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f5ae810f9d0 con 0x7f5ad8069360 2024-02-20T07:58:54.137 INFO:teuthology.orchestra.run.smithi055.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager.smithi055 smithi055 *:9093,9094 running (5m) 55s ago 6m 24.8M - 0.25.0 c8568f914cd2 78ec2fe9d5fe 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi055 smithi055 running (7m) 55s ago 7m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 cb7d541d7455 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi186 smithi186 running (6m) 61s ago 6m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 bd75f3a6c061 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:grafana.smithi055 smithi055 *:3000 running (5m) 55s ago 6m 86.1M - 9.4.7 954c08fa6188 064f316099b8 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi055.ptjwai smithi055 running (80s) 55s ago 79s 10.5M - 17.2.7-521-gc80e7168 10b0c032c913 be2f3aca6608 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi055.sinapd smithi055 running (73s) 55s ago 73s 13.4M - 17.2.7-521-gc80e7168 10b0c032c913 6c3811a23489 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi186.fpwbmz smithi186 running (71s) 61s ago 70s 13.5M - 17.2.7-521-gc80e7168 10b0c032c913 b49044321394 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi186.xntwaj smithi186 running (77s) 61s ago 77s 16.7M - 17.2.7-521-gc80e7168 10b0c032c913 23b02a46c3c4 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi055.stjper smithi055 *:9283 running (8m) 55s ago 8m 480M - 17.2.7-521-gc80e7168 10b0c032c913 eaf5dc05d1a0 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi186.shtgck smithi186 *:8443,9283 running (5m) 61s ago 5m 434M - 17.2.7-521-gc80e7168 10b0c032c913 26e8086a7249 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi055 smithi055 running (8m) 55s ago 8m 46.4M 2048M 17.2.7-521-gc80e7168 10b0c032c913 a759d641dbe8 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi186 smithi186 running (5m) 61s ago 5m 37.8M 2048M 17.2.7-521-gc80e7168 10b0c032c913 6b7e709ad8c3 2024-02-20T07:58:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi055 smithi055 *:9100 running (7m) 55s ago 7m 15.5M - 1.5.0 0da6a335fe13 fd6cd5796b28 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi186 smithi186 *:9100 running (5m) 61s ago 5m 16.0M - 1.5.0 0da6a335fe13 e7382d1097fe 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 smithi055 running (5m) 55s ago 5m 49.0M 4289M 17.2.7-521-gc80e7168 10b0c032c913 cbfa0b8c0da1 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 smithi055 running (4m) 55s ago 4m 47.9M 4289M 17.2.7-521-gc80e7168 10b0c032c913 eb82fe3e8a01 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 smithi055 running (4m) 55s ago 4m 44.2M 4289M 17.2.7-521-gc80e7168 10b0c032c913 4203cbabc127 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 smithi186 running (3m) 61s ago 3m 43.7M 5312M 17.2.7-521-gc80e7168 10b0c032c913 d24330bb95af 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4 smithi186 running (3m) 61s ago 3m 41.9M 5312M 17.2.7-521-gc80e7168 10b0c032c913 a8c7cce19d0f 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5 smithi186 running (2m) 61s ago 2m 42.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 7583a0fcf7c0 2024-02-20T07:58:54.139 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus.smithi055 smithi055 *:9095 running (5m) 55s ago 6m 56.9M - 2.43.0 a07b618ecd1d d66a8225e95d 2024-02-20T07:58:54.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5ad8069360 msgr2=0x7f5ad806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5ad8069360 0x7f5ad806b810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40043d0 tx=0x7f5ad4009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:54.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 msgr2=0x7f5ae810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 0x7f5ae810d960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5adc002a50 tx=0x7f5adc0046b0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 shutdown_connections 2024-02-20T07:58:54.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f5ad8069360 0x7f5ad806b810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae80fbe40 0x7f5ae810d960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 --2- 172.21.15.55:0/3438576514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ae80fcfb0 0x7f5ae810dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 >> 172.21.15.55:0/3438576514 conn(0x7f5ae80f74e0 msgr2=0x7f5ae80656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:54.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 shutdown_connections 2024-02-20T07:58:54.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.139+0000 7f5aef17d700 1 -- 172.21.15.55:0/3438576514 wait complete. 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.294+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/4213797218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 msgr2=0x7ff5e00ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.294+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/4213797218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00ffe60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0004940 tx=0x7ff5d0009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.294+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/4213797218 shutdown_connections 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.294+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/4213797218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.294+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/4213797218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00ffe60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.294+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/4213797218 >> 172.21.15.55:0/4213797218 conn(0x7ff5e00f74e0 msgr2=0x7ff5e00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/4213797218 shutdown_connections 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/4213797218 wait complete. 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 Processor -- start 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 -- start start 2024-02-20T07:58:54.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e00fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e01115a0 con 0x7ff5e00fda80 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5e4d2a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0111710 con 0x7ff5e01003a0 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5de59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.295+0000 7ff5ddd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e00fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5de59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00fd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48846/0 (socket says 172.21.15.55:48846) 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ddd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e00fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.55:49752/0 (socket says 172.21.15.55:49752) 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5de59c700 1 -- 172.21.15.55:0/1082677075 learned_addr learned my addr 172.21.15.55:0/1082677075 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ddd9b700 1 -- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 msgr2=0x7ff5e00fd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ddd9b700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00fd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ddd9b700 1 -- 172.21.15.55:0/1082677075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0005040 con 0x7ff5e01003a0 2024-02-20T07:58:54.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5de59c700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-20T07:58:54.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ddd9b700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e00fb990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009fd0 tx=0x7ff5d400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d4004e90 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e00fbf90 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e00fc4d0 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.296+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5d400f040 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.297+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d400ca50 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.298+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e0107a50 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.299+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff5d400cbb0 con 0x7ff5e01003a0 2024-02-20T07:58:54.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.299+0000 7ff5ceffd700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff5c80690a0 0x7ff5c806b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:54.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.299+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff5d4086b80 con 0x7ff5e01003a0 2024-02-20T07:58:54.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.299+0000 7ff5de59c700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff5c80690a0 0x7ff5c806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:54.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.300+0000 7ff5de59c700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff5c80690a0 0x7ff5c806b550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0013710 tx=0x7ff5d0009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:54.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.308+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff5d404dbd0 con 0x7ff5e01003a0 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: Upgrade: Target is version 19.0.0-1587-g91ff50e9 (unknown) 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024'] 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: Upgrade: Need to upgrade myself (mgr.smithi055.stjper) 2024-02-20T07:58:54.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:54 smithi055 ceph-mon[29254]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:54.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.677+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff5e004bed0 con 0x7ff5e01003a0 2024-02-20T07:58:54.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.678+0000 7ff5ceffd700 1 -- 172.21.15.55:0/1082677075 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7ff5d4056770 con 0x7ff5e01003a0 2024-02-20T07:58:54.679 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "mon": { 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": { 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": { 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "mds": { 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-20T07:58:54.680 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:58:54.681 INFO:teuthology.orchestra.run.smithi055.stdout: "overall": { 2024-02-20T07:58:54.681 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 14 2024-02-20T07:58:54.681 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-02-20T07:58:54.681 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-02-20T07:58:54.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff5c80690a0 msgr2=0x7ff5c806b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff5c80690a0 0x7ff5c806b550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0013710 tx=0x7ff5d0009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 msgr2=0x7ff5e00fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e00fb990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009fd0 tx=0x7ff5d400b040 comp rx=0 tx=0).stop 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 shutdown_connections 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff5c80690a0 0x7ff5c806b550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e00fda80 0x7ff5e00fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 --2- 172.21.15.55:0/1082677075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e01003a0 0x7ff5e00fb990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 >> 172.21.15.55:0/1082677075 conn(0x7ff5e00f74e0 msgr2=0x7ff5e00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 shutdown_connections 2024-02-20T07:58:54.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.680+0000 7ff5e4d2a700 1 -- 172.21.15.55:0/1082677075 wait complete. 2024-02-20T07:58:54.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 -- 172.21.15.55:0/1635389661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 msgr2=0x7f43bc0ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 --2- 172.21.15.55:0/1635389661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 0x7f43bc0ffb30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f43b8004940 tx=0x7f43b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 -- 172.21.15.55:0/1635389661 shutdown_connections 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 --2- 172.21.15.55:0/1635389661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 --2- 172.21.15.55:0/1635389661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 0x7f43bc0ffb30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 -- 172.21.15.55:0/1635389661 >> 172.21.15.55:0/1635389661 conn(0x7f43bc0f74e0 msgr2=0x7f43bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 -- 172.21.15.55:0/1635389661 shutdown_connections 2024-02-20T07:58:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.786+0000 7f43c430d700 1 -- 172.21.15.55:0/1635389661 wait complete. 2024-02-20T07:58:54.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.787+0000 7f43c430d700 1 Processor -- start 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.787+0000 7f43c430d700 1 -- start start 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.787+0000 7f43c430d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 0x7f43bc10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.787+0000 7f43c430d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.787+0000 7f43c430d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc10e470 con 0x7f43bc0663a0 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c430d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc10ed10 con 0x7f43bc100100 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c20a9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 0x7f43bc10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.55:49770/0 (socket says 172.21.15.55:49770) 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 -- 172.21.15.55:0/2487382957 learned_addr learned my addr 172.21.15.55:0/2487382957 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 -- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 msgr2=0x7f43bc10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 0x7f43bc10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 -- 172.21.15.55:0/2487382957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b8005040 con 0x7f43bc100100 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43c18a8700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc10de70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f43ac002a50 tx=0x7f43ac0046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:54.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.788+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43ac007cc0 con 0x7f43bc100100 2024-02-20T07:58:54.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.789+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43ac00f040 con 0x7f43bc100100 2024-02-20T07:58:54.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.789+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43ac00db30 con 0x7f43bc100100 2024-02-20T07:58:54.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.789+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43bc10ef90 con 0x7f43bc100100 2024-02-20T07:58:54.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.789+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43bc10f480 con 0x7f43bc100100 2024-02-20T07:58:54.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.790+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43bc04bed0 con 0x7f43bc100100 2024-02-20T07:58:54.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.790+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f43ac00dc90 con 0x7f43bc100100 2024-02-20T07:58:54.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.791+0000 7f43b2ffd700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f43a806d6b0 0x7f43a806fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:58:54.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.791+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f43ac086ec0 con 0x7f43bc100100 2024-02-20T07:58:54.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.791+0000 7f43c20a9700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f43a806d6b0 0x7f43a806fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:58:54.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.792+0000 7f43c20a9700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f43a806d6b0 0x7f43a806fb60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f43b8004f90 tx=0x7f43b8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:58:54.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:54.799+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f43ac056b10 con 0x7f43bc100100 2024-02-20T07:58:54.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: Upgrade: Target is version 19.0.0-1587-g91ff50e9 (unknown) 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024'] 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: Upgrade: Need to upgrade myself (mgr.smithi055.stjper) 2024-02-20T07:58:54.825 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:54 smithi186 ceph-mon[36772]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:55.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.012+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43bc100d60 con 0x7f43a806d6b0 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.013+0000 7f43b2ffd700 1 -- 172.21.15.55:0/2487382957 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f43bc100d60 con 0x7f43a806d6b0 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024", 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout: "in_progress": true, 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout: "services_complete": [], 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout: "progress": "0/2 daemons upgraded", 2024-02-20T07:58:55.014 INFO:teuthology.orchestra.run.smithi055.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024 image on host smithi186", 2024-02-20T07:58:55.015 INFO:teuthology.orchestra.run.smithi055.stdout: "is_paused": false 2024-02-20T07:58:55.015 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f43a806d6b0 msgr2=0x7f43a806fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f43a806d6b0 0x7f43a806fb60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f43b8004f90 tx=0x7f43b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 msgr2=0x7f43bc10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc10de70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f43ac002a50 tx=0x7f43ac0046b0 comp rx=0 tx=0).stop 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 shutdown_connections 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f43a806d6b0 0x7f43a806fb60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43bc0663a0 0x7f43bc10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 --2- 172.21.15.55:0/2487382957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43bc100100 0x7f43bc10de70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 >> 172.21.15.55:0/2487382957 conn(0x7f43bc0f74e0 msgr2=0x7f43bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 shutdown_connections 2024-02-20T07:58:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:58:55.016+0000 7f43c430d700 1 -- 172.21.15.55:0/2487382957 wait complete. 2024-02-20T07:58:55.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:55 smithi055 ceph-mon[29254]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024 on smithi186 2024-02-20T07:58:55.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:55 smithi055 ceph-mon[29254]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:55.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:55 smithi055 ceph-mon[29254]: from='client.? 172.21.15.55:0/1082677075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-20T07:58:55.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:55 smithi186 ceph-mon[36772]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024 on smithi186 2024-02-20T07:58:55.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:55 smithi186 ceph-mon[36772]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:55.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:55 smithi186 ceph-mon[36772]: from='client.? 172.21.15.55:0/1082677075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-20T07:58:56.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:56 smithi055 ceph-mon[29254]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:56.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:56 smithi055 ceph-mon[29254]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:56.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:56 smithi186 ceph-mon[36772]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:58:56.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:56 smithi186 ceph-mon[36772]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-20T07:58:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:58 smithi055 ceph-mon[29254]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-20T07:58:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:58:58 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:58:58.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:58 smithi186 ceph-mon[36772]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-20T07:58:58.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:58:58 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:59:00.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:00 smithi055 ceph-mon[29254]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-20T07:59:00.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:00 smithi186 ceph-mon[36772]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-20T07:59:02.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:02 smithi186 ceph-mon[36772]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:59:02.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:02 smithi055 ceph-mon[29254]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-20T07:59:04.376 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:04 smithi186 ceph-mon[36772]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-20T07:59:04.385 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:04 smithi055 ceph-mon[29254]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr:Note: switching to '3d72bcfd7affbb4a923c191d212814b4934cfeaa'. 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr:state without impacting any branches by switching back to a branch. 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.528 INFO:tasks.workunit.client.1.smithi186.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: git switch -c 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr:Or undo this operation with: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: git switch - 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr: 2024-02-20T07:59:04.529 INFO:tasks.workunit.client.1.smithi186.stderr:HEAD is now at 3d72bcfd7af test 2024-02-20T07:59:04.533 DEBUG:teuthology.orchestra.run.smithi186:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-20T07:59:04.595 INFO:tasks.workunit.client.1.smithi186.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-20T07:59:04.597 INFO:tasks.workunit.client.1.smithi186.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-20T07:59:04.597 INFO:tasks.workunit.client.1.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-20T07:59:05.429 INFO:tasks.workunit.client.1.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-20T07:59:05.514 INFO:tasks.workunit.client.1.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-20T07:59:05.552 INFO:tasks.workunit.client.1.smithi186.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-20T07:59:05.554 INFO:tasks.workunit.client.1.smithi186.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-20T07:59:05.554 INFO:tasks.workunit.client.1.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-20T07:59:05.594 INFO:tasks.workunit.client.1.smithi186.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-20T07:59:05.598 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T07:59:05.598 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-20T07:59:05.655 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-20T07:59:05.656 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-20T07:59:05.657 DEBUG:teuthology.orchestra.run.smithi186:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d72bcfd7affbb4a923c191d212814b4934cfeaa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-20T07:59:05.719 INFO:tasks.workunit.client.1.smithi186.stderr:+ mkdir -p fsstress 2024-02-20T07:59:05.721 INFO:tasks.workunit.client.1.smithi186.stderr:+ pushd fsstress 2024-02-20T07:59:05.722 INFO:tasks.workunit.client.1.smithi186.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-20T07:59:05.722 INFO:tasks.workunit.client.1.smithi186.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-20T07:59:06.307 INFO:tasks.workunit.client.1.smithi186.stderr:+ tar xzf ltp-full.tgz 2024-02-20T07:59:06.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:06 smithi055 ceph-mon[29254]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-20T07:59:06.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:06 smithi186 ceph-mon[36772]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-20T07:59:06.886 INFO:tasks.workunit.client.0.smithi055.stderr:Note: switching to '3d72bcfd7affbb4a923c191d212814b4934cfeaa'. 2024-02-20T07:59:06.886 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.886 INFO:tasks.workunit.client.0.smithi055.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-20T07:59:06.886 INFO:tasks.workunit.client.0.smithi055.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr:state without impacting any branches by switching back to a branch. 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: git switch -c 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr:Or undo this operation with: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: git switch - 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr: 2024-02-20T07:59:06.887 INFO:tasks.workunit.client.0.smithi055.stderr:HEAD is now at 3d72bcfd7af test 2024-02-20T07:59:06.892 DEBUG:teuthology.orchestra.run.smithi055:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-20T07:59:06.915 INFO:tasks.workunit.client.0.smithi055.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-20T07:59:06.917 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-20T07:59:06.917 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-20T07:59:07.739 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-20T07:59:07.828 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-20T07:59:07.868 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-20T07:59:07.869 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-20T07:59:07.869 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-20T07:59:07.911 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-20T07:59:07.914 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T07:59:07.915 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-20T07:59:07.970 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-20T07:59:07.971 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-20T07:59:07.972 DEBUG:teuthology.orchestra.run.smithi055:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d72bcfd7affbb4a923c191d212814b4934cfeaa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-20T07:59:08.033 INFO:tasks.workunit.client.0.smithi055.stderr:+ mkdir -p fsstress 2024-02-20T07:59:08.036 INFO:tasks.workunit.client.0.smithi055.stderr:+ pushd fsstress 2024-02-20T07:59:08.036 INFO:tasks.workunit.client.0.smithi055.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-20T07:59:08.037 INFO:tasks.workunit.client.0.smithi055.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-20T07:59:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:08 smithi055 ceph-mon[29254]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-20T07:59:08.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:08 smithi186 ceph-mon[36772]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-20T07:59:08.989 INFO:tasks.workunit.client.0.smithi055.stderr:+ tar xzf ltp-full.tgz 2024-02-20T07:59:10.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:10 smithi055 ceph-mon[29254]: pgmap v227: 65 pgs: 65 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 277 KiB/s wr, 0 op/s 2024-02-20T07:59:10.828 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:10 smithi186 ceph-mon[36772]: pgmap v227: 65 pgs: 65 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 277 KiB/s wr, 0 op/s 2024-02-20T07:59:12.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:12 smithi186 ceph-mon[36772]: pgmap v228: 65 pgs: 65 active+clean; 8.2 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 660 KiB/s wr, 10 op/s 2024-02-20T07:59:12.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:12 smithi055 ceph-mon[29254]: pgmap v228: 65 pgs: 65 active+clean; 8.2 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 660 KiB/s wr, 10 op/s 2024-02-20T07:59:14.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:14 smithi055 ceph-mon[29254]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:59:14.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:14 smithi055 ceph-mon[29254]: pgmap v229: 65 pgs: 65 active+clean; 17 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s wr, 78 op/s 2024-02-20T07:59:14.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:14 smithi186 ceph-mon[36772]: from='mgr.14216 172.21.15.55:0/1619392146' entity='mgr.smithi055.stjper' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:59:14.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:14 smithi186 ceph-mon[36772]: pgmap v229: 65 pgs: 65 active+clean; 17 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s wr, 78 op/s 2024-02-20T07:59:16.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:16 smithi055 ceph-mon[29254]: pgmap v230: 65 pgs: 65 active+clean; 34 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s wr, 249 op/s 2024-02-20T07:59:16.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:16 smithi186 ceph-mon[36772]: pgmap v230: 65 pgs: 65 active+clean; 34 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s wr, 249 op/s 2024-02-20T07:59:17.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:17 smithi055 ceph-mon[29254]: fsmap cephfs:2 {0=cephfs.smithi055.sinapd=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:59:17.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:17 smithi186 ceph-mon[36772]: fsmap cephfs:2 {0=cephfs.smithi055.sinapd=up:active,1=cephfs.smithi186.xntwaj=up:active} 2 up:standby 2024-02-20T07:59:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:18 smithi055 ceph-mon[29254]: pgmap v231: 65 pgs: 65 active+clean; 63 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.5 MiB/s wr, 512 op/s 2024-02-20T07:59:18.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:18 smithi186 ceph-mon[36772]: pgmap v231: 65 pgs: 65 active+clean; 63 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.5 MiB/s wr, 512 op/s 2024-02-20T07:59:20.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:20 smithi055 ceph-mon[29254]: pgmap v232: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.9 MiB/s wr, 557 op/s 2024-02-20T07:59:20.824 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:20 smithi186 ceph-mon[36772]: pgmap v232: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.9 MiB/s wr, 557 op/s 2024-02-20T07:59:22.574 INFO:journalctl@ceph.mon.smithi186.smithi186.stdout:Feb 20 07:59:22 smithi186 ceph-mon[36772]: pgmap v233: 65 pgs: 65 active+clean; 85 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.1 MiB/s wr, 557 op/s 2024-02-20T07:59:22.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 07:59:22 smithi055 ceph-mon[29254]: pgmap v233: 65 pgs: 65 active+clean; 85 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.1 MiB/s wr, 557 op/s 2024-02-20T07:59:23.216 INFO:tasks.workunit.client.1.smithi186.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-20T07:59:23.216 INFO:tasks.workunit.client.1.smithi186.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-20T07:59:23.217 INFO:tasks.workunit.client.1.smithi186.stderr:+ make 2024-02-20T07:59:23.238 INFO:tasks.workunit.client.1.smithi186.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-20T07:59:25.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 -- 172.21.15.55:0/823795857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c0fda80 msgr2=0x7fc86c0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 --2- 172.21.15.55:0/823795857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c0fda80 0x7fc86c0ffe60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc85c004940 tx=0x7fc85c009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:25.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 -- 172.21.15.55:0/823795857 shutdown_connections 2024-02-20T07:59:25.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 --2- 172.21.15.55:0/823795857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc86c1003a0 0x7fc86c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 --2- 172.21.15.55:0/823795857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c0fda80 0x7fc86c0ffe60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 -- 172.21.15.55:0/823795857 >> 172.21.15.55:0/823795857 conn(0x7fc86c0f74e0 msgr2=0x7fc86c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 -- 172.21.15.55:0/823795857 shutdown_connections 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.137+0000 7fc87274f700 1 -- 172.21.15.55:0/823795857 wait complete. 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc87274f700 1 Processor -- start 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc87274f700 1 -- start start 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc87274f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc86c0fda80 0x7fc86c0fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc87274f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 0x7fc86c0fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc87274f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0fc020 con 0x7fc86c1003a0 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc87274f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0fc190 con 0x7fc86c0fda80 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc86b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 0x7fc86c0fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc86b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 0x7fc86c0fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40384/0 (socket says 172.21.15.55:40384) 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.138+0000 7fc86b7fe700 1 -- 172.21.15.55:0/849040514 learned_addr learned my addr 172.21.15.55:0/849040514 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc86b7fe700 1 -- 172.21.15.55:0/849040514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc86c0fda80 msgr2=0x7fc86c0fd510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc86b7fe700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc86c0fda80 0x7fc86c0fd510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc86b7fe700 1 -- 172.21.15.55:0/849040514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c005040 con 0x7fc86c1003a0 2024-02-20T07:59:25.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc86b7fe700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 0x7fc86c0fb990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc85400b580 tx=0x7fc8540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:25.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc85400f6f0 con 0x7fc86c1003a0 2024-02-20T07:59:25.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc85400fd20 con 0x7fc86c1003a0 2024-02-20T07:59:25.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc854011ea0 con 0x7fc86c1003a0 2024-02-20T07:59:25.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c0fc470 con 0x7fc86c1003a0 2024-02-20T07:59:25.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.139+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c115a10 con 0x7fc86c1003a0 2024-02-20T07:59:25.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.140+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c04bed0 con 0x7fc86c1003a0 2024-02-20T07:59:25.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.141+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc85400f850 con 0x7fc86c1003a0 2024-02-20T07:59:25.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.141+0000 7fc868ff9700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc8580692a0 0x7fc85806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.141+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc854086dd0 con 0x7fc86c1003a0 2024-02-20T07:59:25.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.141+0000 7fc86bfff700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc8580692a0 0x7fc85806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:25.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.141+0000 7fc86bfff700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc8580692a0 0x7fc85806b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00b3b0 tx=0x7fc85c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:25.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.146+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc854056be0 con 0x7fc86c1003a0 2024-02-20T07:59:25.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.356+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc86c115cb0 con 0x7fc8580692a0 2024-02-20T07:59:25.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.357+0000 7fc868ff9700 1 -- 172.21.15.55:0/849040514 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fc86c115cb0 con 0x7fc8580692a0 2024-02-20T07:59:25.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc8580692a0 msgr2=0x7fc85806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc8580692a0 0x7fc85806b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00b3b0 tx=0x7fc85c009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:25.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 msgr2=0x7fc86c0fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 0x7fc86c0fb990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc85400b580 tx=0x7fc8540091a0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 shutdown_connections 2024-02-20T07:59:25.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc86c0fda80 0x7fc86c0fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fc8580692a0 0x7fc85806b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 --2- 172.21.15.55:0/849040514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc86c1003a0 0x7fc86c0fb990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 >> 172.21.15.55:0/849040514 conn(0x7fc86c0f74e0 msgr2=0x7fc86c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:25.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 shutdown_connections 2024-02-20T07:59:25.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.359+0000 7fc87274f700 1 -- 172.21.15.55:0/849040514 wait complete. 2024-02-20T07:59:25.370 INFO:teuthology.orchestra.run.smithi055.stdout:true 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/2888188314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fbe40 msgr2=0x7fb2a40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/2888188314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fbe40 0x7fb2a40fc250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb294004940 tx=0x7fb294009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/2888188314 shutdown_connections 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/2888188314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/2888188314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fbe40 0x7fb2a40fc250 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/2888188314 >> 172.21.15.55:0/2888188314 conn(0x7fb2a40f7520 msgr2=0x7fb2a40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:25.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.462+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/2888188314 shutdown_connections 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/2888188314 wait complete. 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 Processor -- start 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 -- start start 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a40fbe40 0x7fb2a40fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40fea10 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a9f5c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40feb80 con 0x7fb2a40fbe40 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.463+0000 7fb2a2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40394/0 (socket says 172.21.15.55:40394) 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a2ffd700 1 -- 172.21.15.55:0/4255078401 learned_addr learned my addr 172.21.15.55:0/4255078401 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a2ffd700 1 -- 172.21.15.55:0/4255078401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a40fbe40 msgr2=0x7fb2a40fff00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a2ffd700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a40fbe40 0x7fb2a40fff00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a2ffd700 1 -- 172.21.15.55:0/4255078401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb294005040 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a2ffd700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fe380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb29800b580 tx=0x7fb2980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:25.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb29800f6f0 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb29800fd20 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a40fee60 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb298011ea0 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.464+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a40ff1e0 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.465+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a404bed0 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.466+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb29800f850 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.466+0000 7fb28bfff700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692b0 0x7fb28c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.466+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb298086e70 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.466+0000 7fb2a37fe700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692b0 0x7fb28c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:25.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.467+0000 7fb2a37fe700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692b0 0x7fb28c06b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb294004f90 tx=0x7fb294009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:25.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.471+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb298056be0 con 0x7fb2a40fcfb0 2024-02-20T07:59:25.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.680+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2a4066660 con 0x7fb28c0692b0 2024-02-20T07:59:25.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.681+0000 7fb28bfff700 1 -- 172.21.15.55:0/4255078401 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fb2a4066660 con 0x7fb28c0692b0 2024-02-20T07:59:25.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692b0 msgr2=0x7fb28c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692b0 0x7fb28c06b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb294004f90 tx=0x7fb294009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 msgr2=0x7fb2a40fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fe380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb29800b580 tx=0x7fb2980091a0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 shutdown_connections 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2a40fbe40 0x7fb2a40fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7fb28c0692b0 0x7fb28c06b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 --2- 172.21.15.55:0/4255078401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2a40fcfb0 0x7fb2a40fe380 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 >> 172.21.15.55:0/4255078401 conn(0x7fb2a40f7520 msgr2=0x7fb2a4065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 shutdown_connections 2024-02-20T07:59:25.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.683+0000 7fb2a9f5c700 1 -- 172.21.15.55:0/4255078401 wait complete. 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 -- 172.21.15.55:0/1015781372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 msgr2=0x7f81440ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 --2- 172.21.15.55:0/1015781372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f81440ffe60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8134009ce0 tx=0x7f8134004d10 comp rx=0 tx=0).stop 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 -- 172.21.15.55:0/1015781372 shutdown_connections 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 --2- 172.21.15.55:0/1015781372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81441003a0 0x7f8144102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 --2- 172.21.15.55:0/1015781372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f81440ffe60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 -- 172.21.15.55:0/1015781372 >> 172.21.15.55:0/1015781372 conn(0x7f81440f7500 msgr2=0x7f81440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 -- 172.21.15.55:0/1015781372 shutdown_connections 2024-02-20T07:59:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 -- 172.21.15.55:0/1015781372 wait complete. 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 Processor -- start 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.787+0000 7f81498da700 1 -- start start 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f81498da700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f814410d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f81498da700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81441003a0 0x7f814410dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f81498da700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814410e570 con 0x7f81440fda80 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f81498da700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814410edb0 con 0x7f81441003a0 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f814410d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f814410d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40412/0 (socket says 172.21.15.55:40412) 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 -- 172.21.15.55:0/102028263 learned_addr learned my addr 172.21.15.55:0/102028263 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 -- 172.21.15.55:0/102028263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81441003a0 msgr2=0x7f814410dee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81441003a0 0x7f814410dee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:25.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 -- 172.21.15.55:0/102028263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8134009940 con 0x7f81440fda80 2024-02-20T07:59:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f8142ffd700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f814410d9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8134007bb0 tx=0x7f8134004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f813400e050 con 0x7f81440fda80 2024-02-20T07:59:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814410f030 con 0x7f81440fda80 2024-02-20T07:59:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f813400be70 con 0x7f81440fda80 2024-02-20T07:59:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81340129c0 con 0x7f81440fda80 2024-02-20T07:59:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.788+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814410f570 con 0x7f81440fda80 2024-02-20T07:59:25.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.789+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814404bed0 con 0x7f81440fda80 2024-02-20T07:59:25.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.790+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8134019040 con 0x7f81440fda80 2024-02-20T07:59:25.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.791+0000 7f813b7fe700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f81300692a0 0x7f813006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:25.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.791+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8134088f70 con 0x7f81440fda80 2024-02-20T07:59:25.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.791+0000 7f81427fc700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f81300692a0 0x7f813006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:25.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.791+0000 7f81427fc700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f81300692a0 0x7f813006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f812c009740 tx=0x7f812c006c60 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:25.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:25.795+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f81340babd0 con 0x7f81440fda80 2024-02-20T07:59:26.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.000+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f814410fa10 con 0x7f81300692a0 2024-02-20T07:59:26.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.009+0000 7f813b7fe700 1 -- 172.21.15.55:0/102028263 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f814410fa10 con 0x7f81300692a0 2024-02-20T07:59:26.010 INFO:teuthology.orchestra.run.smithi055.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-20T07:59:26.010 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager.smithi055 smithi055 *:9093,9094 running (5m) 87s ago 7m 24.8M - 0.25.0 c8568f914cd2 78ec2fe9d5fe 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi055 smithi055 running (7m) 87s ago 7m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 cb7d541d7455 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi186 smithi186 running (6m) 93s ago 6m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 bd75f3a6c061 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:grafana.smithi055 smithi055 *:3000 running (5m) 87s ago 7m 86.1M - 9.4.7 954c08fa6188 064f316099b8 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi055.ptjwai smithi055 running (112s) 87s ago 111s 10.5M - 17.2.7-521-gc80e7168 10b0c032c913 be2f3aca6608 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi055.sinapd smithi055 running (105s) 87s ago 105s 13.4M - 17.2.7-521-gc80e7168 10b0c032c913 6c3811a23489 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi186.fpwbmz smithi186 running (103s) 93s ago 102s 13.5M - 17.2.7-521-gc80e7168 10b0c032c913 b49044321394 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi186.xntwaj smithi186 running (109s) 93s ago 109s 16.7M - 17.2.7-521-gc80e7168 10b0c032c913 23b02a46c3c4 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi055.stjper smithi055 *:9283 running (8m) 87s ago 8m 480M - 17.2.7-521-gc80e7168 10b0c032c913 eaf5dc05d1a0 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi186.shtgck smithi186 *:8443,9283 running (6m) 93s ago 6m 434M - 17.2.7-521-gc80e7168 10b0c032c913 26e8086a7249 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi055 smithi055 running (8m) 87s ago 9m 46.4M 2048M 17.2.7-521-gc80e7168 10b0c032c913 a759d641dbe8 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi186 smithi186 running (6m) 93s ago 6m 37.8M 2048M 17.2.7-521-gc80e7168 10b0c032c913 6b7e709ad8c3 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi055 smithi055 *:9100 running (7m) 87s ago 7m 15.5M - 1.5.0 0da6a335fe13 fd6cd5796b28 2024-02-20T07:59:26.011 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi186 smithi186 *:9100 running (6m) 93s ago 6m 16.0M - 1.5.0 0da6a335fe13 e7382d1097fe 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 smithi055 running (5m) 87s ago 5m 49.0M 4289M 17.2.7-521-gc80e7168 10b0c032c913 cbfa0b8c0da1 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 smithi055 running (5m) 87s ago 5m 47.9M 4289M 17.2.7-521-gc80e7168 10b0c032c913 eb82fe3e8a01 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 smithi055 running (4m) 87s ago 4m 44.2M 4289M 17.2.7-521-gc80e7168 10b0c032c913 4203cbabc127 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 smithi186 running (4m) 93s ago 4m 43.7M 5312M 17.2.7-521-gc80e7168 10b0c032c913 d24330bb95af 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4 smithi186 running (3m) 93s ago 3m 41.9M 5312M 17.2.7-521-gc80e7168 10b0c032c913 a8c7cce19d0f 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5 smithi186 running (3m) 93s ago 3m 42.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 7583a0fcf7c0 2024-02-20T07:59:26.012 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus.smithi055 smithi055 *:9095 running (5m) 87s ago 6m 56.9M - 2.43.0 a07b618ecd1d d66a8225e95d 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f81300692a0 msgr2=0x7f813006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f81300692a0 0x7f813006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f812c009740 tx=0x7f812c006c60 comp rx=0 tx=0).stop 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 msgr2=0x7f814410d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f814410d9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8134007bb0 tx=0x7f8134004d10 comp rx=0 tx=0).stop 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 shutdown_connections 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7f81300692a0 0x7f813006b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81440fda80 0x7f814410d9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 --2- 172.21.15.55:0/102028263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81441003a0 0x7f814410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 >> 172.21.15.55:0/102028263 conn(0x7f81440f7500 msgr2=0x7f81440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:26.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.011+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 shutdown_connections 2024-02-20T07:59:26.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.012+0000 7f81498da700 1 -- 172.21.15.55:0/102028263 wait complete. 2024-02-20T07:59:26.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 -- 172.21.15.55:0/3653956224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 msgr2=0x7ff1280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 --2- 172.21.15.55:0/3653956224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff1280fd420 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff110004940 tx=0x7ff110009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:26.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 -- 172.21.15.55:0/3653956224 shutdown_connections 2024-02-20T07:59:26.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 --2- 172.21.15.55:0/3653956224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff1280fd420 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 --2- 172.21.15.55:0/3653956224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280fbe40 0x7ff1280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 -- 172.21.15.55:0/3653956224 >> 172.21.15.55:0/3653956224 conn(0x7ff1280f7520 msgr2=0x7ff1280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:26.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 -- 172.21.15.55:0/3653956224 shutdown_connections 2024-02-20T07:59:26.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.111+0000 7ff12c870700 1 -- 172.21.15.55:0/3653956224 wait complete. 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff12c870700 1 Processor -- start 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff12c870700 1 -- start start 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff12c870700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280fbe40 0x7ff12810d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff12c870700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff12810df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff12c870700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12810e5b0 con 0x7ff1280fcfb0 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff12c870700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12810edf0 con 0x7ff1280fbe40 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff11dd9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff12810df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff11dd9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff12810df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40420/0 (socket says 172.21.15.55:40420) 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff11dd9b700 1 -- 172.21.15.55:0/3554624542 learned_addr learned my addr 172.21.15.55:0/3554624542 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff11dd9b700 1 -- 172.21.15.55:0/3554624542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280fbe40 msgr2=0x7ff12810d9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff11dd9b700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280fbe40 0x7ff12810d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.112+0000 7ff11dd9b700 1 -- 172.21.15.55:0/3554624542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110005040 con 0x7ff1280fcfb0 2024-02-20T07:59:26.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.113+0000 7ff11dd9b700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff12810df20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff110004890 tx=0x7ff110009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.113+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11000e030 con 0x7ff1280fcfb0 2024-02-20T07:59:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.113+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff110004290 con 0x7ff1280fcfb0 2024-02-20T07:59:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.113+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12810f070 con 0x7ff1280fcfb0 2024-02-20T07:59:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.113+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11001c660 con 0x7ff1280fcfb0 2024-02-20T07:59:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.113+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12810f5b0 con 0x7ff1280fcfb0 2024-02-20T07:59:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.114+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12804bed0 con 0x7ff1280fcfb0 2024-02-20T07:59:26.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.114+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff11000bbf0 con 0x7ff1280fcfb0 2024-02-20T07:59:26.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.115+0000 7ff11f7fe700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff1140692a0 0x7ff11406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:26.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.115+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff110088c70 con 0x7ff1280fcfb0 2024-02-20T07:59:26.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.115+0000 7ff12659c700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff1140692a0 0x7ff11406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:26.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.115+0000 7ff12659c700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff1140692a0 0x7ff11406b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff118009740 tx=0x7ff118006c60 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:26.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.119+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff110058960 con 0x7ff1280fcfb0 2024-02-20T07:59:26.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.515+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff12810fa00 con 0x7ff1280fcfb0 2024-02-20T07:59:26.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.516+0000 7ff11f7fe700 1 -- 172.21.15.55:0/3554624542 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7ff1100584f0 con 0x7ff1280fcfb0 2024-02-20T07:59:26.517 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-02-20T07:59:26.517 INFO:teuthology.orchestra.run.smithi055.stdout: "mon": { 2024-02-20T07:59:26.517 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-20T07:59:26.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": { 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": { 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "mds": { 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "overall": { 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 14 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-02-20T07:59:26.518 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-02-20T07:59:26.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff1140692a0 msgr2=0x7ff11406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff1140692a0 0x7ff11406b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff118009740 tx=0x7ff118006c60 comp rx=0 tx=0).stop 2024-02-20T07:59:26.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 msgr2=0x7ff12810df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff12810df20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff110004890 tx=0x7ff110009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 shutdown_connections 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280fbe40 0x7ff12810d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff1140692a0 0x7ff11406b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 --2- 172.21.15.55:0/3554624542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1280fcfb0 0x7ff12810df20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 >> 172.21.15.55:0/3554624542 conn(0x7ff1280f7520 msgr2=0x7ff128065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 shutdown_connections 2024-02-20T07:59:26.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.518+0000 7ff12c870700 1 -- 172.21.15.55:0/3554624542 wait complete. 2024-02-20T07:59:26.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.624+0000 7ff9d81ed700 1 -- 172.21.15.55:0/1733417766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 msgr2=0x7ff9d00fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.624+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/1733417766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d00fd420 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0004940 tx=0x7ff9c0009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:26.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.624+0000 7ff9d81ed700 1 -- 172.21.15.55:0/1733417766 shutdown_connections 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.624+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/1733417766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d00fd420 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.624+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/1733417766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d00fbe40 0x7ff9d00fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.624+0000 7ff9d81ed700 1 -- 172.21.15.55:0/1733417766 >> 172.21.15.55:0/1733417766 conn(0x7ff9d00f74e0 msgr2=0x7ff9d00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 -- 172.21.15.55:0/1733417766 shutdown_connections 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 -- 172.21.15.55:0/1733417766 wait complete. 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 Processor -- start 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 -- start start 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d00fbe40 0x7ff9d010da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d010df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d010e5e0 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d81ed700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d010ee20 con 0x7ff9d00fbe40 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d5788700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d010df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:26.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d5788700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d010df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40444/0 (socket says 172.21.15.55:40444) 2024-02-20T07:59:26.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.625+0000 7ff9d5788700 1 -- 172.21.15.55:0/618018873 learned_addr learned my addr 172.21.15.55:0/618018873 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-02-20T07:59:26.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9d5788700 1 -- 172.21.15.55:0/618018873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d00fbe40 msgr2=0x7ff9d010da10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-20T07:59:26.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9d5788700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d00fbe40 0x7ff9d010da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9d5788700 1 -- 172.21.15.55:0/618018873 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0005040 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9d5788700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d010df50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0008da0 tx=0x7ff9c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9c000e070 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d010f0a0 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d010f580 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9c0009430 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.626+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9c001c400 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.627+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d004bed0 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.628+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff9c000bbf0 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.628+0000 7ff9c67fc700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff9bc0692b0 0x7ff9bc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:59:26.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.628+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff9c0088950 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.628+0000 7ff9d5f89700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff9bc0692b0 0x7ff9bc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:59:26.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.629+0000 7ff9d5f89700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff9bc0692b0 0x7ff9bc06b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009740 tx=0x7ff9cc006c60 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:59:26.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.634+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff9c00586c0 con 0x7ff9d00fcfb0 2024-02-20T07:59:26.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.901+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 --> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9d0065dd0 con 0x7ff9bc0692b0 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.902+0000 7ff9c67fc700 1 -- 172.21.15.55:0/618018873 <== mgr.14216 v2:172.21.15.55:6800/762822871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ff9d0065dd0 con 0x7ff9bc0692b0 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024", 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "in_progress": true, 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "services_complete": [], 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "progress": "0/2 daemons upgraded", 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024 image on host smithi186", 2024-02-20T07:59:26.903 INFO:teuthology.orchestra.run.smithi055.stdout: "is_paused": false 2024-02-20T07:59:26.904 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-02-20T07:59:26.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff9bc0692b0 msgr2=0x7ff9bc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff9bc0692b0 0x7ff9bc06b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009740 tx=0x7ff9cc006c60 comp rx=0 tx=0).stop 2024-02-20T07:59:26.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 msgr2=0x7ff9d010df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:59:26.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d010df50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0008da0 tx=0x7ff9c0009e30 comp rx=0 tx=0).stop 2024-02-20T07:59:26.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 shutdown_connections 2024-02-20T07:59:26.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d00fbe40 0x7ff9d010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:6800/762822871,v1:172.21.15.55:6801/762822871] conn(0x7ff9bc0692b0 0x7ff9bc06b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 --2- 172.21.15.55:0/618018873 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9d00fcfb0 0x7ff9d010df50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:59:26.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 >> 172.21.15.55:0/618018873 conn(0x7ff9d00f74e0 msgr2=0x7ff9d00656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:59:26.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 shutdown_connections 2024-02-20T07:59:26.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T07:59:26.904+0000 7ff9d81ed700 1 -- 172.21.15.55:0/618018873 wait complete. 2024-02-20T07:59:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:45.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:45.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:45.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:45.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:45.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:45.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:46.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:46.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:46.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:46.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:46.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:46.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:47.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:47.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:47.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:47.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:47.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:47.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:47.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:48.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:48.328+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:48.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:48.328+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:48.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:48.328+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:48.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:48.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:48.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:48.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:48.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:48.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:49.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:49.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:49.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:49.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:49.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:49.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:49.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:49.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:49.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:49.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:49.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:50.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:50.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:50.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:50.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:50.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:50.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:50.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:50.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:50.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:50.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:50.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:50.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:50.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:50.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:50.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:51.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:51.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:51.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:51.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:51.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:51.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:51.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:51.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:52.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:51.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:52.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:51.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:52.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:51.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:52.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:52.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:52.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:52.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:52.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:52.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:52.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:52.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:52.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:52.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:52.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:52.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:52.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:52.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:52.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:53.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:53.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:53.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:53.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:53.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:53.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:53.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:53.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:53.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:53.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:53.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:53.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:54.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:53.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:54.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:53.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:54.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:53.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:54.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:54.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:54.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:54.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:54.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:54.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:54.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:54.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:54.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:55.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:55.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:55.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:55.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:55.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:55.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:55.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:55.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:56.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:56.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:56.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:55.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:56.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:55.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:56.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:55.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:56.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:56.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:56.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:56.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:56.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:57.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:56.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:57.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:56.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:57.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:56.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:57.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:57.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:57.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:57.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:57.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:57.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:57.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:57.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:57.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:58.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:58.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:58.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:58.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:58.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:58.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:58.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:58.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:58.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:58.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:58.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:58.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T07:59:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:59.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:59.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T07:59:59.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T07:59:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:59.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:59.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T07:59:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T07:59:59.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:00.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:59.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:00.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:59.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:00.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 07:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T07:59:59.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:00.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:00.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:00.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:00.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:00.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:00.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:01.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:00.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:01.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:00.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:01.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:00.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:01.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:01.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:01.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:01.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:01.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:01.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:01.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:01.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:01.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:02.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:02.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:02.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:02.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:02.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:02.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:02.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:02.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:02.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:02.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:02.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:02.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:02.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:02.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:02.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:03.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:03.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:03.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:03.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:03.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:03.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:03.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:03.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:03.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:03.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:03.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:03.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:03.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:03.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:03.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:04.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:04.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:04.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:04.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:04.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:04.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:04.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:04.241+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:04.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:04.241+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:04.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:04.241+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:04.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:04.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:04.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:04.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:04.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:04.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:05.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:05.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:05.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:05.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:05.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:05.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:05.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:05.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:05.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:05.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:05.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:05.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:05.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:05.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:05.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:06.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:06.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:06.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:06.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:06.241+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:06.241+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:06.241+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:07.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:06.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:07.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:06.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:06.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:07.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:07.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:07.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:07.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:07.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:07.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:07.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:08.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:07.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:08.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:07.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:08.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:07.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:08.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:08.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:08.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:08.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:08.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:08.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:08.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:08.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:08.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:08.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:08.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:08.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:08.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:08.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:08.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:09.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:09.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:09.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:09.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:09.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:09.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:09.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:09.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:09.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:09.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:09.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:09.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:09.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:09.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:09.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:10.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:10.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:10.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:10.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:10.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:10.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:10.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:10.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:10.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:10.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:10.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:10.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:10.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:10.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:10.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:11.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:11.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:11.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:11.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:11.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:11.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:11.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:11.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:11.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:11.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:11.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:11.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:11.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:11.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:11.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:12.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:12.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:12.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:12.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:12.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:12.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:12.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:12.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:12.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:13.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:12.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:13.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:12.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:13.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:12.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:13.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:13.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:13.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:13.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:13.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:13.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:13.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:13.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:13.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:13.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:13.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:13.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:13.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:13.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:13.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:14.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:14.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:14.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:14.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:14.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:14.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:14.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:14.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:14.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:14.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:14.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:14.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:14.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:14.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:14.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:15.670 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:15.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:15.671 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:15.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:15.671 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:15.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:15.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:15.363+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:15.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:15.363+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:15.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:15.363+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:15.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:15.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:15.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:16.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:16.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:16.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:16.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:16.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:16.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:16.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:16.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:16.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:16.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:16.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:16.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:16.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:16.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:16.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:17.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:17.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:17.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:17.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:17.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:17.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:17.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:17.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:17.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:17.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:18.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:18.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:18.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:18.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:18.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:18.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:18.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:18.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:18.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:18.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:18.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:18.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:19.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:19.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:19.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:19.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:19.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:19.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:19.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:19.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:19.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:19.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:19.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:19.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:19.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:19.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:19.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:20.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:20.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:20.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:20.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:20.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:20.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:20.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:20.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:20.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:21.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:20.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:21.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:20.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:21.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:20.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:21.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:21.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:21.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:21.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:21.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:21.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:22.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:21.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:22.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:21.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:22.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:21.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:22.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:22.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:22.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:22.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:22.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:22.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:22.700 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:22.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:22.700 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:22.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:22.700 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:22.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:22.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:22.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:22.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:23.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:23.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:23.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:23.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:23.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:23.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:23.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:23.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:23.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:24.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:24.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:24.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:24.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:24.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:24.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:24.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:24.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:24.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:24.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:24.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:24.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:24.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:24.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:24.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:25.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:25.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:25.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:25.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:25.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:25.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:25.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:25.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:25.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:25.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:25.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:25.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:25.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:25.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:25.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:26.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:26.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:26.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:26.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:26.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:26.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:26.659 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:26.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:26.659 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:26.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:26.659 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:26.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:26.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:26.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:26.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:27.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:27.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:27.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:27.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:27.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:27.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:27.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:27.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:27.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:27.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:27.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:27.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:27.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:27.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:27.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:28.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:28.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:28.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:28.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:28.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:28.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:28.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:28.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:28.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:28.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:28.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:29.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:29.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:29.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:29.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:29.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:29.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:29.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:29.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:29.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:29.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:29.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:29.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:29.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:29.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:30.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:30.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:30.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:30.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:30.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:30.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:30.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:30.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:30.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:31.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:31.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:31.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:31.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:31.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:31.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:31.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:31.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:31.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:31.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:31.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:31.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:32.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:32.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:32.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:32.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:32.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:32.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:32.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:32.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:32.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:32.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:32.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:32.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:32.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:32.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:32.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:33.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:33.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:33.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:33.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:33.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:33.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:33.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:33.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:33.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:33.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:33.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:33.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:34.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:34.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:34.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:34.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:34.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:34.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:34.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:34.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:34.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:34.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:34.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:34.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:35.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:35.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:35.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:35.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:35.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:35.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:35.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:35.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:35.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:35.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:35.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:35.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:36.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:36.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:36.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:36.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:36.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:36.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:37.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:36.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:37.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:36.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:36.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:36.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:36.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:36.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:37.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:37.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:37.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:38.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:37.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:38.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:37.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:38.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:37.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:38.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:37.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:38.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:37.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:38.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:37.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:38.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:38.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:38.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:38.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:38.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:38.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:38.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:38.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:38.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:38.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:38.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:38.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:39.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:39.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:39.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:40.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:39.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:40.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:39.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:40.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:39.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:39.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:39.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:39.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:40.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:40.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:40.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:40.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:40.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:40.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:40.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:40.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:40.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:40.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:40.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:40.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:41.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:41.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:41.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:41.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:41.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:41.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:41.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:41.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:41.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:41.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:41.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:41.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:42.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:42.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:42.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:43.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:42.760+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:43.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:42.760+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:43.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:42.760+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:43.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:42.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:43.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:42.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:43.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:42.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:43.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:43.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:43.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:43.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:43.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:43.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:43.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:43.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:43.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:43.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:43.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:43.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:44.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:44.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:44.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:44.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:44.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:44.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:44.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:44.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:44.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:45.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:45.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:45.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:45.790+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:46.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:45.790+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:46.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:45.790+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:45.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:45.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:45.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:46.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:46.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:46.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:47.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:46.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:47.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:46.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:46.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:46.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:46.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:46.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:47.468+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:47.468+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:47.468+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:48.085 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:47.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:48.085 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:47.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:48.085 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:47.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:48.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:47.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:48.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:47.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:48.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:47.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:48.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:48.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:48.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:48.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:48.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:48.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:48.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:48.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:48.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:48.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:48.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:48.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:48.793+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:48.793+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:48.793+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:49.766 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:49.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:49.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:49.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:49.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:49.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:49.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:49.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:49.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:49.806+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:49.806+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:49.806+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:50.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:50.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:50.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:50.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:50.806 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:50.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:50.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:50.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:50.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:50.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:50.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:50.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:50.804+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:50.804+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:50.804+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:51.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:51.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:51.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:51.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:51.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:51.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:51.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:51.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:51.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:52.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:52.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:52.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:52.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:52.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:52.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:53.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:52.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:53.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:52.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:53.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:52.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:53.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:52.793+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:53.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:52.793+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:53.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:52.793+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:53.750 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:53.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:53.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:53.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:53.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:53.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:53.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:53.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:53.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:53.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:53.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:53.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:54.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:54.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:54.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:54.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:54.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:54.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:54.779+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:54.779+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:54.779+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:54.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:54.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:54.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:55.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:55.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:55.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:55.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:55.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:55.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:55.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:55.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:55.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:55.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:55.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:56.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:56.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:56.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:56.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:56.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:56.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:56.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:56.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:56.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:56.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:56.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:57.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:57.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:57.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:57.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:57.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:57.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:57.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:57.761+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:57.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:57.761+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:57.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:57.761+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:58.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:57.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:58.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:57.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:58.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:57.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:58.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:58.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:58.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:58.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:58.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:58.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:58.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:58.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:58.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:59.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:59.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:00:59.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:00:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:59.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:59.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:00:59.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:00:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:59.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:59.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:00:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:00:59.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:00.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:00.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:00.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:00.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:00.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:00.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:00.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:00.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:00.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:00.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:00.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:01.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:01.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:01.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:01.760+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:01.760+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:01.760+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:01.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:01.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:01.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:02.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:02.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:02.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:02.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:02.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:02.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:02.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:02.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:02.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:02.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:02.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:02.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:03.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:03.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:03.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:03.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:03.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:03.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:04.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:03.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:03.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:03.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:03.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:03.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:03.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:05.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:04.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:05.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:04.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:05.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:04.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:05.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:04.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:05.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:04.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:05.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:04.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:05.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:04.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:05.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:04.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:05.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:04.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:05.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:05.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:05.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:05.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:05.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:05.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:05.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:05.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:05.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:06.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:06.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:06.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:06.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:06.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:06.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:06.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:06.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:06.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:08.081 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:07.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:08.081 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:07.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:08.081 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:07.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:08.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:07.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:08.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:07.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:08.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:07.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:08.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:07.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:08.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:07.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:08.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:07.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:08.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:08.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:08.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:08.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:08.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:08.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:08.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:08.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:08.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:09.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:10.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:09.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:10.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:09.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:10.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:09.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:10.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:09.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:10.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:09.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:10.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:09.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:10.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:09.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:10.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:09.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:10.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:10.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:10.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:10.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:10.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:10.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:10.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:10.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:10.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:11.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:11.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:11.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:11.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:11.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:11.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:12.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:11.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:12.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:11.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:12.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:11.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:11.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:11.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:11.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:13.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:12.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:13.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:12.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:13.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:12.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:13.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:12.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:13.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:12.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:13.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:12.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:13.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:12.900+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:13.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:12.900+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:13.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:12.900+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:13.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:13.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:13.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:13.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:13.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:13.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:13.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:13.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:13.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:13.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:13.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:13.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:14.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:14.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:14.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:14.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:14.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:14.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:14.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:14.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:14.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:15.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:15.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:15.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:15.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:15.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:15.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:15.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:15.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:15.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:15.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:15.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:16.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:15.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:16.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:15.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:16.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:15.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:16.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:16.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:16.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:16.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:16.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:16.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:16.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:16.851+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:16.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:16.851+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:16.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:16.851+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:17.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:16.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:17.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:16.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:17.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:16.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:17.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:17.674+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:17.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:17.674+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:17.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:17.674+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:18.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:17.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:18.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:17.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:18.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:17.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:18.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:17.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:18.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:17.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:18.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:17.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:18.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:18.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:18.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:18.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:18.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:18.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:18.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:18.840+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:18.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:18.840+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:18.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:18.840+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:18.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:18.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:18.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:19.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:19.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:19.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:19.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:19.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:19.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:19.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:19.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:19.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:20.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:20.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:20.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:20.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:20.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:20.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:20.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:20.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:20.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:21.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:21.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:21.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:21.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:21.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:21.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:21.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:21.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:21.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:21.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:21.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:21.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:22.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:22.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:22.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:22.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:22.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:22.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:22.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:22.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:22.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:23.833 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:23.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:23.833 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:23.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:23.833 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:23.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:24.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:23.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:24.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:23.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:23.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:23.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:23.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:23.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:24.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:24.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:24.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:24.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:24.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:24.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:24.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:24.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:24.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:24.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:24.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:24.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:24.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:24.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:24.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:25.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:25.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:25.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:25.790+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:25.790+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:25.790+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:25.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:25.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:25.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:26.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:26.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:26.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:26.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:26.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:26.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:26.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:26.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:26.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:27.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:26.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:27.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:27.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:27.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:27.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:27.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:27.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:27.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:27.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:27.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:28.081+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:28.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:28.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:28.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:29.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:28.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:29.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:28.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:29.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:28.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:29.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:28.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:29.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:28.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:29.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:28.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:29.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:29.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:29.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:30.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:29.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:30.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:29.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:30.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:29.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:30.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:29.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:30.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:29.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:30.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:29.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:30.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:30.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:30.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:30.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:30.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:30.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:30.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:30.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:30.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:30.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:30.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:30.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:31.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:31.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:31.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:31.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:31.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:31.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:32.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:32.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:32.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:32.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:32.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:32.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:32.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:32.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:32.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:33.081+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:33.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:33.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:33.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:33.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:33.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:33.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:33.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:33.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:33.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:33.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:34.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:33.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:34.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:33.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:34.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:33.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:34.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:34.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:34.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:34.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:34.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:34.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:34.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:34.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:34.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:35.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:35.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:35.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:35.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:35.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:35.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:35.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:35.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:35.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:35.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:36.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:35.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:36.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:35.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:36.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:36.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:36.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:36.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:36.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:36.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:36.981 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:36.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:36.981 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:36.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:36.981 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:36.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:37.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:36.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:37.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:36.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:37.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:36.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:37.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:37.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:37.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:37.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:37.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:37.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:38.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:37.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:38.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:37.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:38.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:37.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:38.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:37.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:38.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:37.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:38.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:37.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:38.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:38.080+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:38.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:38.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:38.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:38.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:38.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:38.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:38.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:38.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:38.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:38.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:38.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:38.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:38.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:38.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:38.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:39.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:39.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:39.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:39.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:39.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:39.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:39.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:39.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:39.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:40.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:40.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:40.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:40.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:40.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:40.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:40.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:40.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:40.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:41.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:41.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:41.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:41.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:41.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:41.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:41.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:41.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:41.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:41.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:41.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:41.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:41.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:41.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:41.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:42.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:42.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:42.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:42.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:42.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:42.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:42.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:42.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:42.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:42.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:42.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:42.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:42.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:42.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:42.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:42.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:42.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:42.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:43.208 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:43.082+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:43.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:43.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:43.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:43.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:43.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:43.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:43.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:43.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:43.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:43.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:43.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:43.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:43.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:43.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:43.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:44.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:44.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:44.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:44.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:44.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:44.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:44.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:45.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:44.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:45.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:44.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:45.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:44.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:45.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:45.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:45.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:45.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:45.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:45.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:45.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:45.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:45.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:45.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:45.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:45.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:46.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:45.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:46.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:45.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:46.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:45.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:46.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:46.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:46.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:46.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:46.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:46.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:46.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:46.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:46.650+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:47.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:47.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:47.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:47.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:47.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:47.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:47.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:47.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:47.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:47.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:47.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:47.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:47.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:47.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:47.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:48.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:48.082+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:47.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:47.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:47.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:48.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:48.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:48.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:48.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:48.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:48.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:49.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:49.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:49.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:49.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:49.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:49.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:49.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:49.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:49.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:49.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:49.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:49.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:49.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:49.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:49.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:49.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:49.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:49.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:50.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:50.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:50.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:50.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:50.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:50.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:50.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:50.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:50.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:50.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:50.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:50.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:51.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:50.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:51.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:50.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:51.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:50.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:51.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:51.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:51.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:51.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:51.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:51.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:51.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:51.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:51.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:51.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:51.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:51.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:51.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:51.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:51.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:52.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:52.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:52.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:52.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:52.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:52.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:52.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:52.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:52.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:52.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:52.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:52.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:53.208 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:53.083+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 98 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:53.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:52.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:53.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:52.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:53.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:52.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:53.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:53.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:53.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:53.711+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:53.711+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:53.711+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:54.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:53.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:54.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:53.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:54.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:53.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:54.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:54.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:54.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:54.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:54.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:54.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:54.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:54.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:54.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:54.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:54.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:54.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:55.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:55.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:55.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:55.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:55.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:55.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:55.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:55.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:55.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:56.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:56.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:56.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:56.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:56.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:56.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:56.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:56.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:56.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:57.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:57.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:57.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:57.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:57.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:57.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:57.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:57.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:57.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:58.207 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:01:58.083+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 130 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:01:58.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:57.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:58.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:57.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:58.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:57.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:58.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:58.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:58.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:58.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:58.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:58.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:58.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:58.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:58.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:01:59.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:59.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:59.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:59.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:59.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:01:59.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:01:59.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:59.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:59.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:59.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:01:59.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:01:59.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:00.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:59.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:00.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:59.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:00.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:01:59.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:00.937 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:00.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:00.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:00.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:00.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:00.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:00.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:00.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:00.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:00.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:00.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:00.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:01.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:00.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:01.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:00.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:01.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:00.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:01.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:01.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:01.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:01.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:01.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:01.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:01.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:01.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:01.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:01.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:01.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:01.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:02.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:01.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:02.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:01.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:01.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:02.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:02.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:02.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:02.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:02.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:02.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:02.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:02.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:02.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:02.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:02.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:02.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:03.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:03.084+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 156 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:03.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:02.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:03.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:02.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:03.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:02.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:03.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:03.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:03.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:03.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:03.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:03.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:03.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:03.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:03.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:03.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:04.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:03.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:04.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:03.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:03.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:04.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:04.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:04.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:04.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:04.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:04.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:04.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:04.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:04.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:04.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:04.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:04.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:04.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:04.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:04.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:05.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:05.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:05.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:05.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:05.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:05.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:05.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:05.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:06.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:05.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:06.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:05.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:05.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:06.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:06.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:06.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:06.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:06.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:06.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:06.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:06.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:06.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:06.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:06.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:06.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:06.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:06.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:06.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:07.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:07.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:07.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:07.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:07.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:07.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:07.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:07.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:07.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:07.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:07.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:07.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:08.209 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:08.084+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 185 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:08.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:07.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:08.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:07.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:08.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:07.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:08.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:08.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:08.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:08.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:08.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:08.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:08.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:08.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:08.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:08.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:08.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:08.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:08.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:08.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:08.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:09.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:09.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:09.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:09.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:09.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:09.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:09.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:09.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:09.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:09.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:09.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:09.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:09.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:09.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:09.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:10.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:10.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:10.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:10.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:10.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:10.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:10.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:10.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:10.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:10.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:10.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:10.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:11.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:10.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:11.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:10.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:11.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:10.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:11.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:11.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:11.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:11.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:11.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:11.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:11.608+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:11.608+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:11.608+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:12.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:12.572+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:12.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:12.572+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:12.880 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:12.572+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:12.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:12.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:12.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:12.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:12.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:12.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:13.207 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:13.085+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 212 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:13.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:12.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:13.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:12.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:13.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:12.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:13.898 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:13.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:13.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:13.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:13.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:13.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:13.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:13.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:13.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:13.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:13.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:13.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:13.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:13.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:13.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:14.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:14.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:14.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:14.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:14.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:14.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:14.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:14.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:14.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:14.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:14.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:14.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:14.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:14.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:14.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:15.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:15.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:15.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:15.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:15.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:15.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:15.870 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:15.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:15.870 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:15.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:15.870 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:15.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:16.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:15.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:16.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:15.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:16.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:15.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:16.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:16.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:16.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:16.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:16.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:16.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:16.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:16.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:16.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:16.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:16.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:16.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:17.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:16.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:17.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:16.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:17.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:16.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:17.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:17.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:17.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:17.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:17.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:17.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:17.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:17.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:17.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:17.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:17.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:17.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:18.210 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:18.085+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 242 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:18.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:17.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:18.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:17.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:18.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:17.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:18.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:18.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:18.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:18.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:18.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:18.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:18.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:18.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:18.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:18.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:18.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:18.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:18.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:18.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:18.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:19.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:19.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:19.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:19.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:19.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:19.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:19.914 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:19.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:19.914 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:19.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:19.915 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:19.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:19.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:19.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:19.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:20.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:20.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:20.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:20.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:20.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:20.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:20.596+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:20.596+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:20.596+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:20.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:20.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:20.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:21.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:21.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:21.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:21.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:21.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:21.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:21.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:21.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:21.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:22.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:22.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:22.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:22.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:22.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:22.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:22.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:23.448 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:23.086+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 268 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:23.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:22.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:23.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:22.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:23.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:22.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:23.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:23.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:23.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:23.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:23.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:23.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:24.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:24.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:24.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:24.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:24.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:24.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:24.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:24.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:24.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:24.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:24.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:24.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:24.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:24.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:24.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:25.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:25.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:25.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:25.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:25.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:25.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:25.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:25.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:25.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:26.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:26.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:26.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:26.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:26.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:26.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:26.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:26.447+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:26.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:26.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:26.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:26.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:27.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:27.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:27.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:27.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:27.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:27.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:27.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:27.454+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:28.210 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:28.085+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 298 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:28.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:27.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:28.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:27.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:27.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:27.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:28.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:28.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:28.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:28.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:28.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:28.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:28.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:28.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:28.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:28.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:28.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:28.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:28.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:29.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:29.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:29.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:29.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:29.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:29.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:29.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:29.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:29.541+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:30.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:29.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:30.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:29.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:29.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:29.937+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:30.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:30.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:30.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:30.480+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:30.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:30.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:30.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:30.559+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:30.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:30.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:30.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:30.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:31.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:31.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:31.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:31.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:31.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:31.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:31.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:31.484+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:31.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:31.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:31.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:31.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:31.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:31.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:31.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:31.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:31.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:32.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:31.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:32.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:31.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:32.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:31.943+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:32.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:32.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:32.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:32.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:32.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:32.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:32.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:32.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:33.087+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 325 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:33.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:32.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:32.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:32.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:32.977+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:33.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:33.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:33.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:33.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:33.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:33.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:33.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:33.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:34.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:33.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:34.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:33.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:34.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:33.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:34.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:33.997+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:34.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:34.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:34.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:34.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:34.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:34.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:34.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:34.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:34.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:34.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:34.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:34.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:35.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:35.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:35.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:35.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:35.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:35.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:35.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:35.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:35.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:35.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:35.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:35.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:35.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:35.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:35.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:35.583+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.007+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:36.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:36.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:36.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:36.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:36.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:36.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:36.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:36.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:37.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:37.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:37.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:37.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:36.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:37.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:37.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:37.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:37.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:37.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:37.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:37.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:37.485+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:38.457 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:38.087+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 355 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:38.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:37.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:38.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:37.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:38.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:37.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:38.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:37.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:38.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:38.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:38.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:38.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:38.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:38.457+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:38.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:38.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:38.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:38.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:38.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:38.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:38.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:38.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:39.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:39.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:39.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:39.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:39.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:39.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:39.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:39.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:40.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:39.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:40.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:39.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:40.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:39.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:40.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:39.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:40.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:40.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:40.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:40.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:40.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:40.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:40.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:40.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:41.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:40.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:41.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:40.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:41.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:40.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:41.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:40.990+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:41.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:41.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:41.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:41.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:41.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:41.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:41.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:41.395+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:41.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:41.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:41.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:41.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:41.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:41.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:41.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:41.962+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:42.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:42.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:42.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:42.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:42.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:42.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:42.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:42.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:43.213 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:43.087+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 384 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:43.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:42.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:43.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:42.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:43.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:42.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:43.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:42.923+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:43.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:43.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:43.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:43.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:43.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:43.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:43.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:43.462+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:43.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:43.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:43.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:43.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:43.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:43.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:43.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:43.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:43.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:43.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:43.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:43.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:44.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:44.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:44.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:44.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:44.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:44.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:44.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:44.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:44.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:44.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:44.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:44.938+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:45.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:45.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:45.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:45.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:45.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:45.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:45.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:45.469+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:46.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:45.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:46.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:45.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:46.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:45.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:46.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:45.988+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:46.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:46.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:46.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:46.463+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:46.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:46.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:46.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:46.395+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:47.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:47.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:47.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:47.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:47.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:47.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:47.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:47.011+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:47.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:47.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:47.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:47.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:47.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:47.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:47.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:47.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:47.502+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:48.340 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:48.088+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 416 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:48.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:48.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:48.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:48.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:48.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:48.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:48.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:48.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:48.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:48.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:48.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:48.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:48.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:48.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:48.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:48.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:48.340+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:49.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:49.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:49.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:49.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.012+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:49.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:49.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:49.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:49.530+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:49.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:49.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:49.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:49.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:49.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:49.328+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:49.977+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:50.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:50.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:50.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:50.517+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:50.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:50.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:50.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:50.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:51.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:50.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:51.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:50.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:51.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:50.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:51.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:50.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:51.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:51.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:51.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:51.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:51.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:51.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:51.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:51.531+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:51.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:51.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:51.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:51.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:51.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:51.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:51.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:51.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:52.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:52.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:52.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:52.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:52.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:52.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:52.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:52.028+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:52.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:52.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:52.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:52.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:52.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:52.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:52.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:52.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:53.393 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:53.088+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 443 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:53.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:53.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:53.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:53.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:53.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:53.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:53.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:53.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:53.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:53.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:53.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:53.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:53.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:53.394+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:53.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:53.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:53.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:53.561+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:54.352 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:54.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:54.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:54.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:54.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:54.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:54.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:54.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:54.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:54.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:54.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:54.554+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:54.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:54.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:54.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:54.352+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:55.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:55.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:55.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:55.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:55.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:55.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:55.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:55.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:55.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:55.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:55.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:55.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:55.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:55.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:55.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:55.318+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:55.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:55.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:55.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:55.601+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:56.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:56.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:56.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:56.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:56.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:56.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:56.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:56.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:56.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:56.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:56.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:56.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:56.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:56.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:56.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:56.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:56.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:56.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:56.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:56.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:57.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:57.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:57.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:57.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:57.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:57.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:57.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:57.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:57.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:57.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:57.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:57.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:57.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:57.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:57.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:57.389+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:57.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:57.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:57.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:57.647+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:58.436 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:02:58.089+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 475 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:02:58.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:58.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:58.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:58.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:58.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:58.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:58.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:58.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:58.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:58.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:58.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:58.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:58.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:58.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:58.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:58.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:59.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:59.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:59.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:59.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:59.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:59.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:02:59.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:02:59.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:02:59.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:59.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:59.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:59.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:59.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:59.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:02:59.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:02:59.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:59.695+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:59.695+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:59.695+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:02:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:02:59.696+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:00.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:00.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:00.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:00.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:00.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:00.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:00.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:00.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:00.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:00.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:00.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:00.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:00.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:00.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:00.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:00.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:00.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:00.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:00.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:00.710+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:01.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:01.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:01.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:01.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:01.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:01.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:01.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:01.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:01.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:01.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:01.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:01.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:01.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:01.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:01.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:01.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:02.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:02.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:02.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:02.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:02.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:02.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:02.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:02.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:02.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:02.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:02.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:02.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:02.701+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:02.701+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:02.701+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:02.702+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:03.429 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:03.089+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 504 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:03.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:03.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:03.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:03.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:03.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:03.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:03.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:03.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:03.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:03.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:03.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:03.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:03.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:03.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:03.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:03.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:03.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:03.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:03.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:03.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:03.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:03.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:03.688+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:04.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:04.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:04.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:04.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:04.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:04.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:04.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:04.076+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:04.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:04.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:04.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:04.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:05.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:04.730+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:05.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:04.730+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:05.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:04.730+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:05.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:04.731+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:05.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:05.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:05.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:05.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:05.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:05.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:05.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:05.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:05.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:05.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:05.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:05.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:06.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:05.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:06.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:05.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:06.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:05.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:06.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:05.737+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:06.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:06.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:06.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:06.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:06.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:06.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:06.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:06.095+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:06.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:06.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:06.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:06.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:07.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:06.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:07.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:06.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:07.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:06.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:07.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:06.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:07.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:07.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:07.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:07.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:07.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:07.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:07.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:07.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:07.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:07.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:07.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:07.678+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:07.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:07.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:07.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:07.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:08.452 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:08.090+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 532 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:08.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:08.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:08.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:08.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:08.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:08.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:08.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:08.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:08.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:08.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:08.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:08.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:09.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:08.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:09.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:08.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:09.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:08.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:09.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:08.727+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:09.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:09.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:09.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:09.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:09.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:09.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:09.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:09.714+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:09.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:09.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:09.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:09.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:10.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:10.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:10.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:10.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:10.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:10.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:10.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:10.717+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:10.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:10.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:10.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:10.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:11.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:11.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:11.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:11.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:11.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:11.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:11.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:11.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:11.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:11.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:11.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:11.514+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:11.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:11.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:11.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:11.767+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:12.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:12.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:12.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:12.091+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:12.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:12.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:12.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:12.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:12.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:12.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:12.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:12.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:13.064 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:12.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:13.064 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:12.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:13.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:12.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:13.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:12.774+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:13.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:13.090+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 560 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:13.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:13.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:13.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:13.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:13.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:13.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:13.064+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:13.822 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:13.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:13.822 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:13.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:13.822 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:13.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:13.822 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:13.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:14.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:14.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:14.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:14.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:13.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:13.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:13.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:13.822+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:14.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:14.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:14.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:14.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:15.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:15.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:15.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:15.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:14.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:14.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:14.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:14.832+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:15.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:15.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:15.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:15.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:15.498+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:16.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:15.791+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:16.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:15.791+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:16.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:15.791+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:16.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:15.792+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:16.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:16.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:16.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:16.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:16.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:16.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:16.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:16.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:16.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:16.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:16.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:16.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:16.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:16.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:16.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:16.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:17.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:17.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:17.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:17.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:17.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:17.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:17.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:17.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:18.113 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:18.089+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 589 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:18.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:18.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:18.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:18.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:18.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:18.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:18.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:18.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:18.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:17.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:18.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:17.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:18.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:17.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:18.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:17.857+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:18.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:18.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:18.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:18.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:18.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:18.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:18.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:18.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:19.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:19.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:19.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:19.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:19.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:19.062+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:18.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:18.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:18.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:18.825+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:19.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:19.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:19.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:19.452+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:20.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:20.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:20.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:20.057+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:19.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:19.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:19.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:19.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:20.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:20.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:20.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:20.421+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:20.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:20.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:20.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:20.873+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:21.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:21.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:21.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:21.042+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:21.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:21.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:21.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:21.455+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:21.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:21.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:21.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:21.894+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:22.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:22.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:22.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:22.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:22.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:22.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:22.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:22.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:23.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:23.091+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 616 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:22.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:22.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:22.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:22.897+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:23.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:23.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:23.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:23.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:23.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:23.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:23.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:23.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:23.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:23.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:23.448+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:23.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:23.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:23.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:23.902+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:24.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:24.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:24.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:24.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:24.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:24.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:24.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:24.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:25.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:25.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:25.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:25.089+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:24.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:24.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:24.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:24.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:25.845 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:25.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:25.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:25.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:25.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:25.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:25.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:25.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:26.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:26.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:26.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:26.085+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:25.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:25.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:25.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:25.845+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:26.844 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:26.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:26.845 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:26.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:26.845 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:26.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:26.845 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:26.499+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:26.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:26.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:26.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:26.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.041+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:27.854 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:27.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:27.854 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:27.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:27.854 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:27.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:27.854 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:27.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:28.114 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:28.091+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 644 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:27.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:27.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:27.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:27.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:28.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:27.854+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:28.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:28.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:28.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:28.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:28.897 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:28.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:28.897 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:28.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:28.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:28.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:28.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:28.896+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.001+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:29.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:29.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:29.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:29.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:29.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:29.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:29.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:29.555+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:30.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:30.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:29.998+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:29.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:29.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:29.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:29.941+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:30.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:30.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:30.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:30.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:30.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:30.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:30.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:30.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:31.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:30.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:31.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:30.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:31.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:30.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:31.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:30.954+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:30.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:30.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:30.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:30.986+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:31.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:31.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:31.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:31.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:31.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:31.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:31.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:31.646+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:31.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:31.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:31.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:31.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:31.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:31.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:31.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:31.982+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:32.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:32.604+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:32.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:32.604+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:32.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:32.604+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:32.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:32.605+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:33.216 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:33.092+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 672 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:33.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:33.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:33.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:33.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:33.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:33.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:33.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:33.008+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:33.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:32.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:33.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:32.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:33.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:32.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:33.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:32.923+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:33.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:33.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:33.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:33.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:33.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:33.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:33.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:33.581+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:33.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:33.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:33.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:33.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:34.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:34.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:34.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:34.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:34.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:34.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:34.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:34.583+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:35.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:34.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:35.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:34.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:34.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:34.876+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:34.985+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:35.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:35.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:35.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:35.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:35.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:35.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:35.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:35.559+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:35.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:35.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:35.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:35.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:35.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:35.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:35.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:35.943+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:36.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:36.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:36.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:36.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:36.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:36.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:36.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:36.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:37.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:36.895+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:37.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:36.895+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:36.895+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:36.896+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:36.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:36.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:36.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:36.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:37.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:37.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:37.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:37.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:37.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:37.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:37.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:37.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:38.215 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:38.092+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 703 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:38.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:38.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:38.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:38.020+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:37.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:37.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:37.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:38.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:37.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:38.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:38.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:38.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:38.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:38.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:38.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:38.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:38.516+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:39.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:39.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:39.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:39.031+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:38.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:38.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:38.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:38.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:39.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:39.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:39.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:39.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:39.509+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:40.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:39.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:39.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:39.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:39.933+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:40.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:40.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:40.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:40.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:40.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:40.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:40.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:40.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:40.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:41.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:41.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:41.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:41.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:41.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:41.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:41.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:41.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:42.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:42.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:42.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:42.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:41.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:41.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:41.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:41.941+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:42.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:42.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:42.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:42.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:43.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:43.093+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 730 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:42.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:42.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:42.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:42.969+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:43.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:43.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:43.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:43.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:43.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:43.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:43.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:43.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:43.431+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:44.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:44.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:44.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:44.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:44.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:44.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:44.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:44.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:44.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:43.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:44.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:43.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:44.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:43.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:44.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:43.987+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:44.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:44.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:44.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:44.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:44.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:44.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:44.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:44.977+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:45.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:45.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:45.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:45.152+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:45.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:45.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:45.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:45.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:45.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:45.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:45.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:45.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:45.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:45.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:45.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:45.967+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:46.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:46.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:46.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:46.185+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:46.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:46.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:46.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:46.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:46.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:46.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:46.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:46.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:46.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:46.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:46.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:46.965+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:47.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:47.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:47.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:47.167+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:47.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:47.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:47.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:47.430+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:48.177 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:48.094+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 773 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:48.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:47.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:48.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:47.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:48.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:47.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:48.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:47.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:48.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:48.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:48.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:48.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:48.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:48.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:48.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:48.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:48.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:48.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:48.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:48.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:48.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:48.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:48.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:48.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:49.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:49.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:49.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:49.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:48.946+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:48.946+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:48.946+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:48.947+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:49.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:49.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:49.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:49.437+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:50.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:50.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:50.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:50.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:49.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:49.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:49.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:49.926+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:50.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:50.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:50.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:50.408+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:51.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:50.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:51.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:50.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:51.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:50.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:51.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:50.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:51.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:51.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:51.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:51.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:51.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:51.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:51.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:51.168+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:51.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:51.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:51.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:51.864+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:52.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:52.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:52.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:52.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:52.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:52.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:52.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:52.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:52.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:52.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:52.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:52.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:53.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:52.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:53.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:52.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:53.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:52.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:53.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:52.840+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:53.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:53.094+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 801 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:53.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:53.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:53.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:53.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:53.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:53.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:53.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:53.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:53.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:53.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:53.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:53.495+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:54.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:53.791+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:54.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:53.791+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:54.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:53.791+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:54.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:53.792+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:54.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:54.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:54.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:54.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:54.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:54.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:54.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:54.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:54.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:54.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:54.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:54.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:55.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:55.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:55.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:55.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:54.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:54.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:54.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:54.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:55.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:55.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:55.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:55.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:55.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:55.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:55.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:55.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:55.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:55.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:55.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:55.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:56.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:56.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:56.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:56.050+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:56.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:56.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:56.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:56.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:56.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:56.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:56.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:56.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:56.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:56.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:56.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:56.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:57.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:57.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:57.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:57.042+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:57.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:57.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:57.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:57.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:58.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:57.809+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:58.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:57.809+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:58.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:57.809+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:58.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:57.810+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:58.428 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:03:58.094+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 830 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:03:58.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:58.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:58.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:58.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:58.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:58.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:58.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:58.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:58.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:58.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:58.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:58.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:58.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:58.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:58.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:03:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:58.807+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:03:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:03:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:59.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:59.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:59.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:03:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:03:59.394+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:59.780+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:59.780+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:59.780+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:03:59.781+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:03:59.996+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:00.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:00.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:00.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:00.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:01.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:00.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:00.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:00.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:00.766+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:01.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:01.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:01.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:01.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:01.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:01.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:01.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:01.040+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:01.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:01.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:01.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:01.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:01.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:01.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:01.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:01.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:01.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:01.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:01.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:01.758+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:02.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:02.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:02.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:02.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:02.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:02.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:02.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:02.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:02.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:02.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:02.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:02.359+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:03.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:02.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:03.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:02.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:03.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:02.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:03.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:02.804+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:03.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:03.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:03.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:03.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:03.391 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:03.095+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 857 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:03.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:03.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:03.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:03.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:03.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:03.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:03.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:03.316+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:03.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:03.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:03.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:03.789+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:03.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:04.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:04.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:04.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:04.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:04.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:04.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:04.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:04.818+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:04.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:04.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:04.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:04.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:05.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:05.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:05.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:05.312+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:05.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:05.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:05.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:05.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:06.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:06.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:06.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:06.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:06.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:06.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:06.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:06.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:06.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:06.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:06.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:06.828+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:07.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:07.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:07.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:07.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:07.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:07.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:07.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:07.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:08.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:07.997+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:08.095+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 885 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:08.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:08.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:08.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:08.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:08.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:08.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:08.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:08.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:08.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:08.788+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:09.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:09.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:09.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:09.036+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:09.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:09.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:09.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:09.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:10.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:10.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:10.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:10.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:09.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:09.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:09.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:09.778+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:10.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:10.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:10.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:10.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:10.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:10.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:10.314+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:11.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:10.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:11.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:10.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:11.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:10.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:11.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:10.765+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:11.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:11.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:11.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:11.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:11.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:11.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:11.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:11.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:11.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:11.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:11.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:11.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:11.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:11.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:11.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:11.362+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:11.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:11.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:11.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:11.733+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:12.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:12.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:12.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:12.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:12.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:12.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:12.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:12.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:12.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:12.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:12.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:12.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:12.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:12.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:12.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:12.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:12.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:12.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:12.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:12.690+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:13.418 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:13.096+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 913 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:13.418 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:13.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:13.418 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:13.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:13.418 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:13.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:13.418 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:13.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:13.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:13.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:13.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:13.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:14.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:13.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:14.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:13.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:14.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:13.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:14.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:13.739+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:14.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:14.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:14.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:14.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:14.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:14.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:14.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:14.085+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:14.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:14.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:14.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:14.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:15.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:14.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:15.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:14.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:15.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:14.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:15.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:14.743+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:15.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:15.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:15.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:15.055+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:15.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:15.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:15.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:15.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:15.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:15.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:15.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:15.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:16.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:15.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:16.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:15.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:16.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:15.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:16.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:15.741+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:16.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:16.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:16.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:16.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:16.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:16.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:16.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:16.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:16.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:16.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:16.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:16.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:17.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:16.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:17.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:16.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:17.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:16.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:17.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:16.785+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:17.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:17.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:17.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:17.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:17.801 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:17.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:17.801 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:17.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:17.801 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:17.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:17.801 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:17.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:17.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:17.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:17.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:17.801+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:18.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:18.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:18.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:18.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:18.037+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:18.452 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:18.097+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 942 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:18.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:18.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:18.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:18.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.009+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:18.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:18.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:18.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:18.826+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:19.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:19.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:19.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:19.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:19.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:19.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:19.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:19.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:19.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:20.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:20.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:20.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:20.374+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:20.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:20.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:20.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:20.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:20.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:20.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:20.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:20.826+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:21.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:21.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:21.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:21.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:21.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:21.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:21.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:21.385+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:21.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:21.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:21.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:21.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:21.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:21.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:21.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:21.790+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:21.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:21.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:21.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:21.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:22.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:22.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:22.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:22.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:23.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:22.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:23.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:22.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:23.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:22.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:23.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:22.786+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:22.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:22.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:22.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:22.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:23.350 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:23.097+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 969 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:23.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:23.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:23.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:23.350+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:23.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:23.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:23.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:23.741+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:23.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:23.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:23.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:23.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:24.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:24.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:24.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:24.377+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:24.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:24.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:24.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:24.757+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:25.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:25.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:25.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:25.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:25.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:25.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:25.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:25.749+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:25.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:26.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:26.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:26.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:26.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:26.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:26.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:26.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:26.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:26.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:26.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:26.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:26.727+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:27.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:26.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:27.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:26.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:27.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:26.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:27.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:26.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:27.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:27.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:27.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:27.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:27.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:27.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:27.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:27.335+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:27.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:27.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:27.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:27.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:27.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:27.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:27.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:27.712+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:28.360 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:28.098+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1003 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:28.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:28.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:28.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:28.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:28.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:28.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:28.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:28.016+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:28.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:28.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:28.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:28.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:28.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:28.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:28.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:28.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:28.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:28.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:28.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:28.711+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:29.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:29.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:29.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:29.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:29.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:29.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:29.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:29.064+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:29.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:29.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:29.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:29.341+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:29.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:29.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:29.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:29.687+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:30.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:30.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:30.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:30.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:30.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:30.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:30.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:30.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:30.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:30.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:30.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:30.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:30.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:30.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:30.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:30.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:30.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:30.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:30.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:30.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:31.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:31.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:31.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:31.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:31.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:31.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:31.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:31.055+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:31.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:31.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:31.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:31.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:31.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:31.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:31.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:31.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:31.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:31.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:31.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:31.682+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:32.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:32.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:32.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:32.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:32.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:32.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:32.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:32.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:32.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:32.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:32.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:32.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:32.674+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:32.674+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:32.674+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:32.675+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:33.377 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:33.098+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1046 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:33.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:33.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:33.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:33.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:33.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:33.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:33.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:33.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:33.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:33.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:33.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:33.377+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:33.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:33.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:33.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:33.699+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:34.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:34.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:34.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:34.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:33.992+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:34.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:34.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:34.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:34.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:34.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:34.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:34.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:34.350+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:34.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:34.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:34.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:34.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:34.703+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:34.703+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:34.703+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:34.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:35.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:35.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:35.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:35.394+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:35.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:35.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:35.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:35.730+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:35.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:35.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:35.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:35.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:36.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:36.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:36.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:36.396+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:36.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:36.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:36.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:36.757+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:36.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:36.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:36.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:36.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:37.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:37.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:37.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:37.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:37.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:37.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:37.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:37.720+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:37.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:37.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:37.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:38.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:37.960+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:38.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:38.099+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1090 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:38.958 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:38.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:38.958 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:38.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:38.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:38.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:38.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:38.721+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:38.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:38.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:38.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:38.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:38.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:38.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:38.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:38.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:38.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:38.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:38.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:39.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:38.958+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:39.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:39.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:39.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:39.720+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:39.720+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:39.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:39.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:39.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:39.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:39.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:40.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:39.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:40.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:39.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:39.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:40.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:39.933+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:40.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:40.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:40.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:40.754+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:40.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:40.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:40.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:40.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:40.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:40.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:40.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:40.980+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:41.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:41.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:41.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:41.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:41.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:41.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:41.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:41.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:42.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:41.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:41.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:41.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:41.803+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:42.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:42.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:42.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:42.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:42.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:42.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:42.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:42.532+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:43.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:42.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:43.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:42.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:43.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:42.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:43.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:42.806+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:43.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:43.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:43.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:43.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:42.970+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:43.099+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1132 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:43.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:43.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:43.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:43.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:43.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:43.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:43.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:43.570+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:43.850+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:43.850+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:43.850+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:43.851+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:43.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:43.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:43.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:43.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:44.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:44.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:44.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:44.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:44.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:44.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:44.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:44.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:44.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:44.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:44.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:44.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:44.965+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:45.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:45.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:45.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:45.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:45.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:45.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:45.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:45.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:45.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:45.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:45.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:45.830+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:46.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:46.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:46.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:46.013+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:46.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:46.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:46.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:46.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:46.781+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:46.781+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:46.781+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:46.781+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:47.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:47.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:47.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:47.000+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:47.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:47.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:47.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:47.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:47.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:47.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:47.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:47.540+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:47.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:47.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:47.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:47.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:47.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:47.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:47.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:47.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:48.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:48.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:48.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:48.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:48.100+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1175 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:48.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:48.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:48.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:48.586+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:48.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:48.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:48.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:48.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:49.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:49.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:49.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:49.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:49.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:49.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:49.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:49.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:49.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:49.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:49.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:49.551+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:50.052 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:49.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:50.052 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:49.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:50.052 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:49.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:50.052 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:49.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:50.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:50.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:50.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:50.052+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:50.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:50.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:50.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:50.556+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:50.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:50.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:50.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:50.766+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:51.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:51.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:51.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:51.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:51.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:51.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:51.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:51.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:51.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:51.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:51.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:51.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:51.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:51.762+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:52.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:52.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:52.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:52.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:52.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:52.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:52.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:52.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:52.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:52.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:52.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:52.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:53.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:52.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:53.102 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:52.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:53.102 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:52.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:53.102 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:52.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:53.100+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1218 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:53.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:53.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:53.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:53.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:53.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:53.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:53.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:53.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:53.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:53.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:53.764+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:53.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:53.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:53.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:53.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:54.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:54.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:54.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:54.126+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:54.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:54.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:54.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:54.556+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:54.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:54.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:54.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:54.767+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:55.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:55.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:55.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:55.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:55.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:55.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:55.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:55.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:55.576+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:55.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:55.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:55.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:55.799+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:56.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:56.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:56.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:56.148+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:56.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:56.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:56.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:56.614+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:56.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:56.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:56.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:56.845+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T08:04:57.008+0000 7f401b225700 0 monclient(hunting): authenticate timed out after 300 2024-02-20T08:04:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-20T08:04:57.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:57.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:57.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:57.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:57.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:57.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:57.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:57.183+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:57.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-20T08:04:57.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:57.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:57.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:57.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:57.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:57.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:57.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:57.621+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:58.036 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T08:04:58.146 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:04:58.100+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1320 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:04:58.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:57.874+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:58.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:57.874+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:58.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:57.874+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:58.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:57.875+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:58.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:58.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:58.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:58.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:58.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:58.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:58.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:58.145+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:58.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:58.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:58.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:58.579+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:59.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:58.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:59.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:58.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:59.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:58.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:59.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:58.835+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:59.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:59.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:59.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:04:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:04:59.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:04:59.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:59.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:59.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:59.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:59.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:59.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:04:59.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:04:59.579+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:59.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:59.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:59.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:04:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:04:59.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:00.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:00.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:00.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:00.085+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:00.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:00.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:00.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:00.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:00.587+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:00.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:00.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:00.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:00.812+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:01.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:01.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:01.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:01.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:01.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:01.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:01.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:01.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:01.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:01.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:01.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:01.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:02.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:02.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:02.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:02.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:02.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:02.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:02.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:01.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:01.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:01.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:01.859+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:02.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:02.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:02.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:02.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:02.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:02.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:02.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:02.527+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:03.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:02.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:03.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:02.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:03.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:02.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:03.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:02.826+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:03.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:03.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:03.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:03.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:03.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:03.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:03.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:03.038+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:03.102+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1412 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:03.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:03.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:03.860 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:03.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:03.860 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:03.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:03.860 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:03.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:03.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:03.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:03.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:03.860+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:04.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:04.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:04.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:04.077+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:04.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:04.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:04.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:04.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:04.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:04.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:04.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:04.526+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:05.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:05.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:05.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:05.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:04.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:04.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:04.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:04.882+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:05.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:05.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:05.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:05.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:05.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:05.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:05.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:05.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:06.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:06.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:06.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:06.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:06.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:06.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:06.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:06.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:05.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:05.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:05.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:05.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:06.926 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:06.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:06.926 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:06.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:06.926 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:06.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:06.926 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:06.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:07.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:07.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:07.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:07.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:07.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:06.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:06.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:06.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:06.926+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:07.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:07.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:07.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:07.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:08.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:08.102+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1506 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:07.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:07.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:07.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:07.882+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:08.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:08.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:08.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:08.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:08.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:08.858 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:08.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:08.858 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:08.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:08.858 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:08.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:08.858 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:08.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:09.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:09.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:09.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:09.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:09.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:08.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:08.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:08.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:08.858+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:09.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:09.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:09.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:09.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:09.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:09.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:09.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:09.821+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:10.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:10.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:10.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:10.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:10.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:10.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:10.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:10.785+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:10.987+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:11.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:11.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:11.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:11.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:11.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:11.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:11.423+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:12.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:12.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:12.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:12.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:12.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:12.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:12.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:12.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:11.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:11.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:11.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:11.816+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:12.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:12.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:12.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:12.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:12.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:12.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:12.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:12.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:13.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:13.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:13.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:13.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:13.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:13.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:13.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:13.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:13.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:12.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:13.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:12.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:13.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:12.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:13.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:12.843+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:13.412 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:13.103+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1599 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:13.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:13.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:13.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:13.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:14.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:13.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:14.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:13.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:14.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:13.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:14.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:13.812+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:14.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:14.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:14.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:14.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:14.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:14.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:14.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:14.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:14.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:14.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:14.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:14.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:14.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:14.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:14.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:15.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:14.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:15.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:14.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:15.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:14.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:15.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:14.775+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:15.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:15.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:15.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:15.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:15.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:15.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:15.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:15.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:16.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:15.781+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:16.106 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:15.781+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:16.106 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:15.781+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:16.106 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:15.781+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:16.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:16.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:16.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:16.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:16.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:16.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:16.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:16.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:16.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:16.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:16.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:16.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:17.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:16.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:17.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:16.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:17.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:16.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:17.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:16.816+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:17.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:17.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:17.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:17.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:17.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:17.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:17.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:17.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:17.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:17.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:17.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:17.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:18.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:17.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:18.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:17.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:18.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:17.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:18.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:17.839+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:18.455 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:18.103+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1693 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:18.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:18.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:18.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:18.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:18.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:18.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:18.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:18.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:18.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:18.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:18.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:18.455+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:19.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:18.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:19.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:18.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:19.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:18.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:19.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:18.861+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:19.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:19.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:19.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:19.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:19.850 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:19.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:19.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:19.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:19.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:20.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:19.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:20.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:19.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:20.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:19.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:20.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:19.851+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:20.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:20.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:20.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:20.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:20.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:20.498+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:20.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:20.498+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:20.897 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:20.498+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:20.897 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:20.499+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:20.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:20.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:20.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:20.896+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:21.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:21.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:21.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:21.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:21.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:21.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:21.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:21.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:21.474+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:21.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:21.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:21.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:21.937+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:22.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:22.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:22.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:22.119+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:22.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:22.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:22.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:22.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:22.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:22.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:22.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:22.486+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:23.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:23.104+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1785 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:22.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:22.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:22.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:22.923+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:23.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:23.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:23.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:23.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:23.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:23.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:23.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:23.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:23.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:23.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:23.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:23.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:23.949+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:23.949+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:23.949+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:23.950+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:24.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:24.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:24.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:24.114+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:24.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:24.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:24.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:24.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:25.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:24.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:25.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:24.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:25.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:24.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:25.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:24.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:25.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:25.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:25.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:25.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:25.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:25.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:25.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:25.092+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:25.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:25.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:25.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:25.400+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:25.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:25.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:25.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:25.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:26.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:26.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:26.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:26.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:26.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:26.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:26.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:26.430+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:26.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:26.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:26.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:26.969+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:27.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:27.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:27.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:27.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:27.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:27.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:27.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:27.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:28.224 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:28.104+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1878 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:28.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:28.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:28.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:28.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:28.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:28.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:28.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:28.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:28.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:27.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:28.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:27.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:28.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:27.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:28.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:27.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:28.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:28.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:28.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:28.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:28.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:28.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:28.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:28.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:29.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:29.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:29.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:29.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:29.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:29.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:29.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:29.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:29.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:29.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:29.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:29.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:30.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:30.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:30.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:30.025+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:30.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:30.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:30.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:30.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:30.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:30.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:30.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:30.976+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:31.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:31.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:31.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:31.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:31.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:31.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:31.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:31.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:31.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:31.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:31.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:31.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:32.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:32.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:32.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:32.026+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:32.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:32.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:32.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:32.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:33.466 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:33.105+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1971 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:33.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:33.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:33.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:33.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:33.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:33.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:33.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:33.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:32.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:32.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:32.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:32.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:33.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:33.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:33.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:33.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:33.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:33.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:33.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:33.466+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:34.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:33.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:34.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:33.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:34.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:33.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:34.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:33.995+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:34.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:34.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:34.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:34.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:34.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:34.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:34.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:34.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:34.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:34.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:34.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:34.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:34.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:34.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:35.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:35.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:35.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:35.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:35.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:35.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:35.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:35.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:35.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:34.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:35.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:34.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:35.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:34.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:35.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:34.987+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:35.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:35.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:35.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:35.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:36.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:36.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:36.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:36.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:36.148+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:36.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:36.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:36.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:36.385+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:37.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:36.994+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:37.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:37.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:37.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:37.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:37.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:37.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:37.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:37.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:37.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:38.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:38.106+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2065 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.036+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:38.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:38.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:38.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:38.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:38.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:38.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:38.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:38.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:38.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:39.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:39.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:39.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:39.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:39.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:39.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:38.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:39.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:39.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:39.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:39.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:40.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:40.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:40.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:40.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:40.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:40.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:40.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:40.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:40.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:40.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:40.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:40.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:40.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:40.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:40.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:40.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:40.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:40.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:40.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:40.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:40.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:41.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:41.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:41.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:41.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:41.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:41.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:41.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:41.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:41.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:41.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:41.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:41.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:41.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:41.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:41.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:41.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:41.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:41.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:42.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:42.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:42.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:42.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:42.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:42.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:42.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:42.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:42.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:42.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:42.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:42.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:42.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:42.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:42.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:42.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:42.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:42.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:42.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:42.364+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:43.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:43.106+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2157 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:43.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:43.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:43.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:43.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:43.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:43.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:43.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:43.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:43.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:43.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:43.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:43.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:43.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:43.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:43.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:43.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:43.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:44.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:44.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:44.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:44.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:44.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:44.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:44.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:44.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:44.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:44.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:44.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:44.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:44.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:44.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:44.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:44.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:45.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:45.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:45.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:45.103+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:45.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:45.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:45.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:45.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:45.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:45.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:45.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:45.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:45.362+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:46.317 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:46.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:46.317 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:46.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:46.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:46.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:46.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:46.145+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:46.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:46.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:46.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:46.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:46.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:46.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:46.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:46.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:46.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:46.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:46.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:46.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:47.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:47.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:47.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:47.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.009+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:47.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:47.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:47.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:47.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:47.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:48.108 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:48.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:48.108 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:48.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:48.108 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:48.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:48.108 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:48.073+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:48.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:48.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:48.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:48.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:47.971+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:48.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:48.107+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2250 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:48.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:48.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:48.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:48.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:49.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:49.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:49.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:49.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:48.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:48.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:48.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:48.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:49.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:49.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:49.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:49.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:49.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:50.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:49.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:49.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:49.954+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:50.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:50.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:50.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:50.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:50.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:50.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:50.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:50.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:51.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:51.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:51.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:51.028+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:50.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:50.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:50.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:50.961+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:51.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:51.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:51.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:51.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:51.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:51.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:51.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:51.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:52.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:52.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:51.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:51.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:51.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:51.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:52.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:52.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:52.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:52.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:53.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:53.108+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2343 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:53.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:52.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:52.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:52.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:52.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:52.993+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:53.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:53.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:53.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:53.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:53.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:53.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:53.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:53.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:53.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:53.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:53.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:54.000+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:53.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:53.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:53.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:53.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:54.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:54.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:54.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:54.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:55.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:55.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:55.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:55.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:55.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:55.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:55.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:55.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:54.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:54.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:54.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:54.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:55.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:55.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:55.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:55.389+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:56.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:56.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:56.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:56.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:55.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:55.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:55.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:55.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:56.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:56.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:56.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:56.396+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.031+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.031+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.031+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.032+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:56.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:56.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:56.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:56.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:57.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:57.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:57.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:57.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:57.994+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:57.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:57.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:57.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:58.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:57.822+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:58.383 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:05:58.108+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2439 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:05:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:58.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:58.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:58.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:58.382+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:58.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:58.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:58.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:05:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:58.820+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:58.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:58.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:58.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:05:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:58.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:05:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:59.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:59.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:59.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:05:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:05:59.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:00.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:59.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:59.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:59.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:05:59.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:59.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:59.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:59.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:00.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:05:59.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:00.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:00.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:00.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:00.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:00.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:00.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:00.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:00.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:00.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:00.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:00.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:00.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:00.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:01.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:01.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:01.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:01.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:02.008 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:01.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:02.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:01.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:02.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:01.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:02.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:01.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:02.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:02.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:02.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:02.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.008+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:02.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:02.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:02.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:02.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:02.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:02.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:02.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:02.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:03.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:03.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:03.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:03.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:02.960+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:03.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:02.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:03.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:02.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:03.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:02.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:03.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:02.765+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:03.109+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2530 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:03.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:03.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:03.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:03.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:03.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:03.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:03.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:03.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:03.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:03.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:03.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:03.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:03.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:03.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:03.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:03.774+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:04.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:04.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:04.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:04.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:04.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:04.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:04.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:04.962+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:04.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:04.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:04.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:04.726+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:05.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:05.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:05.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:05.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:05.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:05.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:05.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:05.925+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:05.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:05.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:05.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:05.729+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:06.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:06.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:06.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:06.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:06.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:06.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:06.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:06.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:06.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:06.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:06.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:06.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:06.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:06.943+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:07.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:07.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:07.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:07.448+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:07.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:07.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:07.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:07.964+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:08.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:07.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:08.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:07.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:08.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:07.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:08.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:07.729+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:08.449 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:08.110+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2624 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:08.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:08.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:08.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:08.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:08.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:08.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:08.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:08.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:08.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:08.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:08.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:08.926+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:08.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:08.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:08.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:08.718+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:09.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:09.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:09.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:09.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:09.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:09.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:09.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:09.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:09.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:09.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:09.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:09.891+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:09.681+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:09.681+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:09.681+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:09.682+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:10.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:10.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:10.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:10.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:10.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:10.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:10.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:10.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:10.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:10.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:10.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:10.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:10.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:10.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:10.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:10.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:11.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:11.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:11.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:11.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:11.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:11.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:11.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:11.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:11.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:11.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:11.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:11.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:11.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:11.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:11.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:11.909+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:12.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:12.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:12.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:12.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:13.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:12.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:12.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:12.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:12.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:12.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:12.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:12.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:13.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:12.725+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:13.375 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:13.110+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2716 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:13.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:13.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:13.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:13.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:13.375+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:13.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:13.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:13.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:13.755+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:13.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:13.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:13.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:13.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:14.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:14.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:14.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:14.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:14.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:14.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:14.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:14.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:14.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:14.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:14.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:14.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:14.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:14.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:14.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:14.708+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:14.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:14.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:14.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:14.961+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:15.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:15.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:15.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:15.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:15.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:15.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:15.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:15.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:15.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:15.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:15.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:15.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:16.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:16.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:16.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:16.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:16.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:16.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:16.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:16.002+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:16.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:16.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:16.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:16.421+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:16.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:16.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:16.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:16.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:17.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:17.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:17.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:17.015+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:17.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:17.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:17.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:17.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:17.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:17.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:17.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:17.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:18.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:18.111+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2812 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:18.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:18.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:18.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:18.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:18.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:18.004+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:18.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:18.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:18.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:18.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:18.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:18.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:18.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:18.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:19.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:19.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:19.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:19.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:19.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:19.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:19.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:19.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:19.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:19.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:19.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:19.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:20.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:20.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:20.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:20.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:20.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:20.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:20.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:20.632+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:20.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:20.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:20.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:20.492+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:21.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:21.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:21.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:21.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:21.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:21.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:21.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:21.130+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:21.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:21.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:21.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:21.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:21.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:21.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:21.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:21.659+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:22.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:22.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:22.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:22.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:22.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:22.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:22.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:22.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:22.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:22.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:22.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:22.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:23.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:23.112+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2903 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:23.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:23.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:23.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:23.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:23.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:23.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:23.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:23.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:23.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:23.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:23.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:23.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:23.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:23.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:23.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:23.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:24.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:24.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:24.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:24.154+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:24.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:24.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:24.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:24.642+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:24.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:24.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:24.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:24.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:25.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:25.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:25.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:25.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:25.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:25.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:25.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:25.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:25.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:25.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:25.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:25.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:25.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:25.457+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:25.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:25.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:25.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:25.631+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:26.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:26.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:26.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:26.147+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:26.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:26.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:26.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:26.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:26.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:26.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:26.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:26.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:27.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:27.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:27.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:27.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:27.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:27.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:27.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:27.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:27.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:27.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:27.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:27.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:27.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:27.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:27.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:27.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:28.463 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:28.112+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2997 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:28.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:28.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:28.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:28.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:28.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:28.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:28.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:28.211+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:28.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:28.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:28.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:28.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:28.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:28.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:28.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:28.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:29.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:29.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:29.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:29.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:29.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:29.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:29.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:29.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:29.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:29.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:29.655+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:29.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:29.655+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:29.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:29.655+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:29.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:29.656+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:30.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:30.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:30.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:30.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:30.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:30.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:30.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:30.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:30.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:30.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:30.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:30.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:30.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:30.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:30.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:30.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:30.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:30.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:31.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:31.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:31.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:31.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:31.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:31.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:31.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:31.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:31.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:31.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:31.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:31.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:31.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:31.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:31.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:31.711+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:32.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:32.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:32.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:32.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:32.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:32.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:32.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:32.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:32.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:32.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:32.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:32.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:32.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:32.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:32.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:32.669+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:33.113+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3089 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:33.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:33.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:33.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:33.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:33.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:33.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:33.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:33.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:33.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:33.532+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:33.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:33.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:33.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:33.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:33.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:33.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:33.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:33.696+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:34.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:34.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:34.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:34.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:34.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:34.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:34.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:34.231+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:34.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:34.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:34.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:34.498+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:34.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:34.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:34.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:34.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:35.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:35.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:35.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:35.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:35.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:35.279+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:35.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:35.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:35.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:35.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:35.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:35.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:35.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:35.511+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:35.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:35.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:35.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:35.645+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:36.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:36.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:36.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:36.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:36.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:36.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:36.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:36.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:36.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:36.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:36.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:36.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:36.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:36.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:37.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:37.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:37.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:37.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:37.270+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:37.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:37.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:37.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:37.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:37.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:37.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:37.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:37.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:37.647+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:38.415 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:38.114+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3184 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:38.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:38.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:38.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:38.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:38.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:38.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:38.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:38.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:38.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:38.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:38.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:38.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:38.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:38.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:38.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:38.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:38.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:38.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:38.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:38.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:39.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:39.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:39.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:39.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:39.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:39.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:39.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:39.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:39.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:39.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:39.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:39.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:39.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:39.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:39.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:39.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:39.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:39.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:39.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:39.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:40.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:40.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:40.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:40.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:40.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:40.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:40.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:40.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:40.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:40.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:40.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:40.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:41.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:40.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:41.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:40.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:41.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:40.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:41.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:40.723+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:41.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:41.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:41.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:41.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:41.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:41.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:41.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:41.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:41.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:41.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:41.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:41.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:41.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:41.763+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:42.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:42.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:42.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:42.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:42.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:42.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:42.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:42.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:43.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:42.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:43.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:42.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:43.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:42.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:43.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:42.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:43.428 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:43.114+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3275 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:43.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:43.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:43.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:43.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:43.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:43.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:43.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:43.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:43.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:43.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:43.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:43.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:43.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:43.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:43.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:44.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:43.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:44.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:44.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:44.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:44.278+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:44.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:44.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:44.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:44.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:44.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:44.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:44.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:44.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:44.718+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:45.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:45.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:45.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:45.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:45.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:45.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:45.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:45.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:45.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:45.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:45.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:45.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:45.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:45.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:45.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:45.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:45.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:45.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:46.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:46.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:46.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:46.362+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:46.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:46.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:46.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:46.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:46.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:46.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:46.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:46.663+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:47.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:47.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:47.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:47.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:47.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:47.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:47.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:47.346+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:47.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:47.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:47.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:47.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:47.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:47.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:47.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:47.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:47.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:47.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:47.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:47.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:47.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:47.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:47.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:47.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:48.394 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:48.115+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3369 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:48.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:48.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:48.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:48.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:48.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:48.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:48.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:48.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:48.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:48.404+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:48.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:48.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:48.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:48.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:48.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:48.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:48.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:48.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:49.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:49.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:49.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:49.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:49.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:49.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:49.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:49.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:49.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:49.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:49.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:49.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:49.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:49.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:49.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:49.623+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:50.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:50.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:50.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:50.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:50.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:50.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:50.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:50.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:50.655+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:50.655+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:50.655+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:50.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:50.655+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:51.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:51.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:51.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:51.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:51.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:51.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:51.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:51.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:51.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:51.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:51.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:51.423+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:52.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:52.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:52.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:52.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:52.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:52.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:52.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:52.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:52.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:52.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:52.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:52.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:52.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:53.435 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:53.116+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3461 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:53.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:53.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:53.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:53.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:53.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:53.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:53.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:53.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:53.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:53.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:53.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:53.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:53.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:54.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:54.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:54.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:54.499+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:54.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:54.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:54.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:54.645+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:54.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:54.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:54.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:54.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:55.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:55.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:55.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:55.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:55.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:55.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:55.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:55.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:55.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:55.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:55.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:55.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:56.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:56.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:56.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:56.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:56.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:56.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:56.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:56.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:56.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:56.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:56.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:56.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:57.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:57.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:57.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:57.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:57.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:57.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:57.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:57.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:57.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:57.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:57.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:57.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:06:58.116+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3557 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:06:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:58.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:58.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:58.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:58.564+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:58.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:58.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:58.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:58.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:58.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:58.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:58.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:58.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:59.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:59.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:59.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:06:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:06:59.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:59.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:59.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:59.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:06:59.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:59.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:59.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:59.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:06:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:06:59.645+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:00.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:00.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:00.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:00.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:00.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:00.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:00.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:00.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:00.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:00.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:00.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:00.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:01.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:01.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:01.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:01.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:01.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:01.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:01.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:01.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:01.685+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:01.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:01.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:01.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:01.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:02.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:02.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:02.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:02.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:02.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:02.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:02.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:02.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:02.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:02.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:02.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:02.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:03.446 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:03.117+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3649 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:03.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:03.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:03.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:03.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:03.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:03.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:03.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:03.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:03.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:03.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:03.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:03.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:04.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:04.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:04.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:04.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:04.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:04.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:04.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:04.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:04.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:04.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:04.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:04.610+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:05.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:05.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:05.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:05.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:05.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:05.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:05.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:05.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:05.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:05.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:05.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:05.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:05.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:05.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:05.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:05.657+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:06.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:06.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:06.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:06.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:06.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:06.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:06.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:06.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:06.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:06.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:06.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:06.692+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:06.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:06.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:06.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:06.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:07.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:07.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:07.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:07.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:07.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:07.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:07.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:07.345+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:07.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:07.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:07.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:07.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:07.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:07.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:07.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:07.712+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:07.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:07.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:07.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:07.739+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:08.118+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3743 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:08.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:08.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:08.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:08.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:08.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:08.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:08.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:08.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:08.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:08.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:08.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:08.728+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:08.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:08.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:08.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:08.702+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:09.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:09.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:09.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:09.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:09.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:09.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:09.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:09.308+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:09.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:09.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:09.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:09.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:09.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:09.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:09.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:09.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:09.768+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:10.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:10.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:10.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:10.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:10.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:10.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:10.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:10.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:10.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:10.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:10.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:10.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:10.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:11.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:11.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:11.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:11.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:11.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:11.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:11.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:11.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:11.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:11.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:11.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:11.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:11.770+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:12.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:12.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:12.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:12.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:12.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:12.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:12.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:12.272+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:12.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:12.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:12.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:12.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:12.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:12.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:12.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:12.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:12.769+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:13.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:13.119+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3835 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:13.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:13.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:13.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:13.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:13.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:13.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:13.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:13.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:13.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:13.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:13.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:13.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:13.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:13.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:13.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:13.595+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:13.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:13.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:13.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:13.768+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:14.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:14.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:14.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:14.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:14.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:14.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:14.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:14.304+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:14.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:14.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:14.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:14.779+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:14.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:14.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:14.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:14.628+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:15.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:15.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:15.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:15.314+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:15.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:15.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:15.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:15.771+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:15.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:16.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:16.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:16.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:16.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:16.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:16.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:16.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:16.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:16.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:16.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:16.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:16.625+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:16.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:16.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:16.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:16.739+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:17.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:17.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:17.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:17.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:17.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:17.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:17.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:17.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:17.591+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:17.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:17.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:17.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:17.734+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:18.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:18.119+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3930 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:18.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:18.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:18.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:18.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:18.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:18.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:18.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:18.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:18.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:18.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:18.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:18.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:18.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:18.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:18.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:18.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:18.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:18.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:18.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:18.712+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:19.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:19.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:19.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:19.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:19.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:19.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:19.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:19.291+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:19.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:19.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:19.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:19.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:19.709+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:19.709+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:19.709+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:19.710+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:20.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:20.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:20.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:20.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:20.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:20.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:20.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:20.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:20.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:20.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:20.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:20.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:20.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:20.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:20.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:20.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:21.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:21.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:21.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:21.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:21.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:21.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:21.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:21.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:21.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:21.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:21.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:21.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:21.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:21.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:21.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:21.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:22.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:22.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:22.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:22.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:22.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:22.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:22.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:22.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:22.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:22.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:22.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:22.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:22.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:22.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:22.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:22.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:23.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:23.120+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4022 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:23.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:23.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:23.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:23.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:23.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:23.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:23.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:23.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:23.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:23.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:23.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:23.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:23.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:23.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:23.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:23.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:23.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:24.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:24.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:24.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:24.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:24.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:24.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:24.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:24.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:24.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:24.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:24.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:24.640+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:24.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:24.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:24.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:24.668+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:25.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:25.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:25.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:25.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:25.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:25.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:25.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:25.592+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:25.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:25.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:25.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:25.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:25.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:25.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:25.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:25.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:25.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:25.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:25.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:25.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:25.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:25.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:25.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:25.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:26.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:26.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:26.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:26.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:26.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:26.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:26.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:26.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:26.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:26.393+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:26.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:26.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:26.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:26.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:27.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:27.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:27.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:27.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:27.625+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:27.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:27.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:27.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:27.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:27.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:27.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:27.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:27.574+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:28.400 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:28.121+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4117 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:28.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:28.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:28.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:28.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:28.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:28.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:28.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:28.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:28.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:28.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:28.400+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:28.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:28.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:28.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:28.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:29.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:29.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:29.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:29.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:29.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:29.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:29.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:29.576+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:29.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:29.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:29.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:29.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:30.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:30.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:30.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:30.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:30.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:30.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:30.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:30.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:30.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:30.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:30.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:30.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:30.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:30.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:30.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:30.461+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:31.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:31.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:31.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:31.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:31.661+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:31.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:31.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:31.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:31.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:31.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:31.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:31.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:31.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:32.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:32.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:32.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:32.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:32.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:32.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:32.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:32.492+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:32.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:32.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:32.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:32.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:32.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:32.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:32.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:32.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:32.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:32.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:32.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:32.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:33.121+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4209 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:33.749+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:33.749+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:33.749+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:33.750+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:33.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:33.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:33.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:33.521+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:33.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:33.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:33.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:33.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:34.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:34.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:34.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:34.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:34.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:34.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:34.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:34.501+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:34.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:34.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:34.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:34.722+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:35.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:35.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:35.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:35.453+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:35.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:35.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:35.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:35.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:35.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:35.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:35.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:35.739+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:36.739 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:36.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:36.739 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:36.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:36.739 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:36.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:36.739 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:36.502+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:36.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:36.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:36.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:36.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:36.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:36.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:36.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:36.738+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:36.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:36.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:36.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:36.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:36.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:36.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:36.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:36.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:37.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:37.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:37.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:37.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:37.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:37.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:37.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:37.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:37.746 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:37.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:37.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:37.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:37.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:37.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:37.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:37.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:38.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:37.745+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:38.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:37.745+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:38.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:37.745+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:38.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:37.746+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:38.454 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:38.122+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4306 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:38.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:38.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:38.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:38.454+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:38.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:38.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:38.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:38.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:38.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:38.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:38.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:38.770+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:39.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:39.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:39.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:39.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:39.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:39.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:39.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:39.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:39.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:39.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:39.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:39.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:39.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:39.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:39.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:39.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:39.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:39.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:39.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:40.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:39.786+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:40.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:40.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:40.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:40.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:40.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:40.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:40.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:40.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:40.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:40.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:40.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:40.810+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:41.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:41.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:41.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:41.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:41.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:41.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:41.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:41.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:41.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:41.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:41.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:41.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:41.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:41.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:41.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:41.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:41.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:41.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:41.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:41.772+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:42.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:42.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:42.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:42.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:42.439+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:42.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:42.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:42.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:42.373+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:43.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:42.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:43.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:42.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:43.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:42.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:43.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:42.818+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:43.393 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:43.123+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4397 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:43.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:43.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:43.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:43.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:43.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:43.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:43.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:43.338+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:43.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:43.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:43.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:43.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:43.393+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:43.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:43.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:43.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:43.863+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:44.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:44.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:44.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:44.391+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:44.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:44.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:44.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:44.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:44.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:44.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:44.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:44.849+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:45.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:45.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:45.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:45.374+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:45.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:45.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:45.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:45.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:45.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:45.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:45.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:45.810+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:46.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:46.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:46.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:46.394+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:46.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:46.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:46.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:46.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:46.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:46.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:46.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:46.797+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:47.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:47.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:47.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:47.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:47.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:47.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:47.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:47.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:47.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:47.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:48.125 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:47.751+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:48.125 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:47.751+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:48.125 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:47.751+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:48.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:47.752+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:48.407 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:48.124+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4498 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:48.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:48.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:48.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:48.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:48.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:48.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:48.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:48.428+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:48.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:48.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:48.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:48.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:49.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:49.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:49.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:49.437+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:49.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:49.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:49.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:49.468+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:50.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:49.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:50.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:49.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:50.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:49.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:50.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:49.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:50.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:50.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:50.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:50.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:50.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:50.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:50.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:50.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:50.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:50.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:50.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:50.471+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:50.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:50.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:50.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:50.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:51.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:51.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:51.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:51.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:51.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:51.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:51.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:51.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:51.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:51.424+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:51.801+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:51.801+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:51.801+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:51.801+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:52.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:52.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:52.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:52.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:52.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:52.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:52.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:52.441+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:53.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:52.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:53.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:52.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:53.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:52.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:53.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:52.763+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:53.390 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:53.125+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4596 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:53.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:53.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:53.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:53.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:53.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:53.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:53.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:53.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:53.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:53.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:53.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:53.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:53.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:53.783+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:54.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:54.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:54.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:54.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:54.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:54.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:54.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:54.443+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:55.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:54.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:55.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:54.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:54.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:54.805+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:55.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:55.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:55.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:55.395+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:55.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:55.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:55.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:55.471+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:55.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:55.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:55.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:55.763+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:56.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:56.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:56.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:56.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:56.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:56.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:56.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:56.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:56.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:56.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:56.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:56.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:57.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:57.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:57.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:57.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:57.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:57.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:57.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:57.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:57.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:57.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:57.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:57.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:57.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:57.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:57.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:57.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:58.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:57.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:58.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:57.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:58.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:57.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:58.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:57.790+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:58.410 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:07:58.125+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4691 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:07:58.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:58.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:58.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:58.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:58.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:58.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:58.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:58.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:58.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:58.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:58.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:58.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:58.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:58.804+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:58.804+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:58.804+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:07:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:58.805+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:07:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:59.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:59.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:59.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:59.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:07:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:07:59.383+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:59.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:59.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:59.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:07:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:07:59.362+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:59.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:59.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:59.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:07:59.775+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:00.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:00.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:00.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:00.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:00.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:00.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:00.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:00.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:00.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:00.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:00.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:00.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:01.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:01.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:01.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:01.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:01.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:01.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:01.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:01.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:01.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:01.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:01.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:01.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:01.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:01.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:01.652 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:01.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:01.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:01.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:01.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:01.748+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:02.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:02.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:02.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:02.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:02.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:02.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:02.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:02.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:02.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:02.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:02.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:02.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:02.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:02.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:02.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:02.237+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:02.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:02.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:02.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:02.718+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:03.431 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:03.126+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4784 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:03.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:03.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:03.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:03.200+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:03.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:03.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:03.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:03.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:03.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:03.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:03.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:03.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:03.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:03.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:03.431+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:04.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:04.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:04.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:04.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:04.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:04.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:04.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:04.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:04.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:04.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:04.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:04.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:04.694+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:05.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:05.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:05.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:05.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:05.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:05.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:05.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:05.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:06.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:05.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:06.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:05.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:06.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:05.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:06.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:05.738+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:06.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:06.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:06.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:06.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:06.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:06.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:06.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:06.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:06.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:06.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:06.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:06.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:06.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:06.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:06.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:06.722+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:07.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:07.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:07.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:07.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:07.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:07.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:07.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:07.398+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:07.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:07.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:07.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:07.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:07.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:07.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:07.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:07.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:07.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:07.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:07.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:07.720+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:08.378 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:08.127+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4893 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:08.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:08.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:08.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:08.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:08.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:08.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:08.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:08.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:08.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:08.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:08.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:08.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:08.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:08.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:08.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:08.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:08.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:08.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:08.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:08.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:08.699+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:09.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:09.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:09.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:09.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:09.384+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:09.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:09.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:09.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:09.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:09.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:09.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:09.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:09.730+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:10.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:10.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:10.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:10.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:10.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:10.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:10.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:10.298+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:10.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:10.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:10.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:10.766+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:11.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:11.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:11.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:11.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:11.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:11.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:11.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:11.249+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:11.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:11.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:11.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:11.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:11.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:11.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:11.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:11.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:11.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:11.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:11.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:11.776+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:12.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:12.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:12.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:12.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:12.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:12.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:12.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:12.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:12.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:12.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:12.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:12.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:12.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:12.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:12.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:12.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:13.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:12.755+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:13.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:12.755+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:13.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:12.755+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:13.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:12.756+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:13.465 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:13.128+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4983 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:13.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:13.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:13.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:13.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:13.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:13.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:13.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:13.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:13.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:13.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:13.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:13.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:13.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:13.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:13.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:13.732+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:14.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:14.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:14.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:14.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:14.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:14.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:14.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:14.466+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:14.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:14.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:14.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:14.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:14.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:14.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:14.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:14.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:15.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:15.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:15.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:15.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:15.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:15.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:15.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:15.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:15.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:15.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:15.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:15.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:15.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:15.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:15.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:16.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:16.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:16.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:16.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:16.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:16.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:16.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:16.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:16.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:16.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:16.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:16.433+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:17.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:17.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:17.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:17.657+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:17.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:17.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:17.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:17.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:17.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:17.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:17.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:17.486+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:18.436 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:18.129+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5080 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:18.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:18.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:18.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:18.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:18.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:18.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:18.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:18.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:18.655+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:18.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:18.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:18.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:18.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:19.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:19.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:19.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:19.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:19.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:19.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:19.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:19.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:19.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:19.675+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:19.675+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:19.675+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:19.676+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:20.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:20.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:20.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:20.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:20.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:20.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:20.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:20.377+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:20.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:20.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:20.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:20.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:20.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:20.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:20.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:20.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:20.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:20.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:20.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:20.631+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:21.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:21.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:21.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:21.607+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:21.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:21.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:21.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:21.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:21.375+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:21.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:21.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:21.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:21.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:22.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:22.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:22.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:22.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:22.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:22.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:22.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:22.353+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:22.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:22.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:22.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:22.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:22.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:22.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:22.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:22.402+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:22.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:22.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:22.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:22.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:22.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:22.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:22.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:22.613+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:23.384 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:23.128+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5192 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:23.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:23.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:23.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:23.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:23.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:23.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:23.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:23.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:23.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:23.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:23.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:23.600+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:23.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:23.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:23.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:23.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:24.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:24.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:24.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:24.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:24.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:24.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:24.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:24.326+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:24.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:24.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:24.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:24.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:24.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:24.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:24.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:24.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:24.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:24.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:24.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:24.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:25.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:25.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:25.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:25.588+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:25.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:25.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:25.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:25.351+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:25.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:25.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:25.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:25.445+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:26.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:26.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:26.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:26.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:26.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:26.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:26.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:26.631+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:26.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:26.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:26.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:26.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:26.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:26.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:26.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:26.483+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:27.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:27.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:27.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:27.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:27.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:27.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:27.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:27.461+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:27.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:27.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:27.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:27.644+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:27.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:27.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:27.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:27.397+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:28.408 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:28.130+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5312 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:28.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:28.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:28.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:28.408+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:28.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:28.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:28.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:28.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:28.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:28.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:28.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:28.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:29.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:29.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:29.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:29.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:29.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:29.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:29.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:29.525+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:29.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:29.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:29.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:29.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:29.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:29.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:29.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:29.393+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:29.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:29.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:29.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:29.654+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:30.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:30.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:30.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:30.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:30.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:30.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:30.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:30.364+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:30.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:30.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:30.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:30.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:30.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:30.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:30.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:30.574+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:30.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:30.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:30.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:30.623+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:31.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:31.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:31.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:31.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:31.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:31.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:31.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:31.384+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:31.652 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:31.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:31.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:31.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:31.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:31.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:31.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:31.535+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:31.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:31.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:31.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:31.668+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:32.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:32.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:32.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:32.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:32.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:32.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:32.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:32.497+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:32.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:32.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:32.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:32.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:32.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:32.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:32.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:32.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:32.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:32.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:32.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:32.669+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:33.131+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5406 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:33.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:33.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:33.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:33.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:33.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:33.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:33.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:33.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:33.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:33.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:33.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:33.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:33.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:33.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:33.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:33.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:34.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:34.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:34.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:34.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:34.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:34.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:34.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:34.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:34.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:34.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:34.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:34.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:34.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:34.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:34.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:34.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:34.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:35.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:35.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:35.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:35.318+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:35.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:35.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:35.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:35.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:35.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:35.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:35.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:35.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:35.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:36.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:36.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:36.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:36.283+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:36.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:36.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:36.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:36.456+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:36.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:36.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:36.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:36.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:37.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:37.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:37.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:37.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:37.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:37.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:37.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:37.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:37.290+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:37.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:37.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:37.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:37.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:38.426 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:38.132+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5502 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:38.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:38.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:38.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:38.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:38.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:38.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:38.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:38.300+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:38.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:38.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:38.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:38.426+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:38.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:38.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:38.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:38.623+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:39.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:39.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:39.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:39.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:39.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:39.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:39.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:39.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:39.623 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:39.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:39.623 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:39.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:39.623 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:39.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:39.623 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:39.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:39.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:39.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:39.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:39.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:40.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:40.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:40.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:40.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:40.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:40.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:40.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:40.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:40.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:40.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:40.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:40.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:40.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:40.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:40.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:40.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:40.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:40.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:40.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:40.672+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:41.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:41.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:41.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:41.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:41.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:41.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:41.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:41.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:41.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:41.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:41.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:41.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:41.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:41.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:41.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:41.416+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:41.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:41.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:41.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:41.668+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:42.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:42.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:42.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:42.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:42.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:42.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:42.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:42.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:42.691 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:42.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:42.691 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:42.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:42.691 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:42.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:42.691 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:42.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:42.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:42.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:42.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:42.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:42.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:42.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:42.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:42.690+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:43.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:43.133+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5593 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:43.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:43.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:43.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:43.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:43.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:43.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:43.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:43.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:43.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:43.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:43.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:43.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:43.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:43.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:43.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:43.507+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:43.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:43.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:43.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:43.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:43.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:43.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:43.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:43.668+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:44.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:44.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:44.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:44.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:44.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:44.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:44.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:44.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:44.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:44.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:44.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:44.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:44.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:44.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:44.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:44.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:44.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:44.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:44.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:44.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:44.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:44.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:44.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:44.643+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:45.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:45.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:45.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:45.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:45.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:45.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:45.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:45.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:45.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:45.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:45.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:45.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:45.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:45.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:45.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:45.569+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:45.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:45.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:45.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:45.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:45.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:45.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:45.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:45.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:46.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:46.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:46.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:46.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:46.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:46.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:46.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:46.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:46.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:46.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:46.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:46.526+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:47.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:47.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:47.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:47.628+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:47.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:47.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:47.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:47.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:47.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:47.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:47.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:47.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:48.134+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5687 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:48.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:48.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:48.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:48.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:48.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:48.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:48.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:48.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:48.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:48.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:48.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:48.561+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:49.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:49.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:49.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:49.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:49.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:49.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:49.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:49.663+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:49.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:49.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:49.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:49.537+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:50.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:50.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:50.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:50.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:50.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:50.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:50.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:50.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:50.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:50.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:50.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:50.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:50.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:50.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:51.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:51.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:51.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:51.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:51.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:51.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:51.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:51.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:51.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:51.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:52.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:51.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:52.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:51.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:52.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:51.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:52.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:51.743+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:52.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:52.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:52.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:52.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:52.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:52.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:52.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:52.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:53.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:52.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:53.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:52.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:53.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:52.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:53.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:52.747+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:53.461 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:53.135+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5781 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:53.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:53.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:53.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:53.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:53.461+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:53.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:53.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:53.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:53.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:53.591+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:53.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:53.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:53.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:53.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:53.718+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:54.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:54.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:54.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:54.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:54.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:54.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:54.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:54.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:54.566+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:54.566+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:54.566+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:54.567+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:55.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:55.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:55.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:55.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:55.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:55.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:55.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:55.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:55.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:55.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:55.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:55.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:55.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:55.747+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:56.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:56.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:56.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:56.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:56.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:56.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:56.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:56.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:56.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:56.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:56.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:56.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:56.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:56.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:56.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:56.588+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:56.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:56.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:56.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:57.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:56.774+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:57.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:57.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:57.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:57.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:57.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:57.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:57.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:57.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:57.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:57.612+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:58.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:57.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:58.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:57.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:58.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:57.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:58.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:57.772+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:08:58.135+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5875 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:08:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:58.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:58.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:58.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:08:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:58.790+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:58.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:58.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:58.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:58.560+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:58.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:58.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:58.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:58.583+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:08:59.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:59.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:59.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:59.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:08:59.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:59.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:59.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:59.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:08:59.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:08:59.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:59.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:59.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:59.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:08:59.787+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:00.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:00.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:00.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:00.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:00.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:00.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:00.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:00.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:00.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:00.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:00.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:00.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:00.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:00.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:00.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:00.825+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:01.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:01.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:01.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:01.532+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:01.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:01.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:01.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:01.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:01.591+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:01.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:01.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:01.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:01.821+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:02.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:02.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:02.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:02.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:02.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:02.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:02.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:02.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:02.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:02.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:02.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:02.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:02.866 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:02.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:02.866 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:02.560+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:03.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:02.864+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:03.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:02.864+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:03.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:02.864+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:03.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:02.865+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:03.136+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5968 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:03.839 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:03.616+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:03.839 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:03.616+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:03.839 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:03.616+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:03.839 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:03.617+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:03.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:03.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:03.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:03.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:03.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:03.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:03.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:03.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:03.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:03.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:03.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:03.839+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:04.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:04.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:04.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:04.626+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:04.554+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:04.554+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:04.554+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:04.554+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:04.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:04.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:04.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:04.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:05.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:05.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:05.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:05.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:05.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:05.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:05.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:05.611+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:05.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:05.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:05.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:05.513+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:05.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:05.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:05.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:05.838+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:06.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:06.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:06.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:06.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:06.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:06.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:06.844 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:06.606+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:06.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:06.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:06.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:06.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:06.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:06.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:06.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:06.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:06.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:06.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:06.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:06.843+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:07.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:07.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:07.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:07.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:07.483+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:07.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:07.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:07.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:07.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:07.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:08.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:07.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:08.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:07.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:08.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:07.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:08.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:07.812+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:08.448 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:08.137+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6063 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:08.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:08.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:08.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:08.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:08.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:08.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:08.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:08.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:08.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:08.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:08.616+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:08.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:08.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:08.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:08.849+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:09.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:09.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:09.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:09.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:09.570+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:09.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:09.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:09.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:09.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:09.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:09.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:09.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:09.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:10.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:10.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:10.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:10.607+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:10.460+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:10.460+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:10.460+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:10.461+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:10.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:10.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:10.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:10.887+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:11.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:11.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:11.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:11.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:11.652+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:11.653+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:11.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:11.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:11.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:11.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:11.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:11.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:11.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:11.424+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:11.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:11.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:11.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:11.850+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:12.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:12.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:12.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:12.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:12.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:12.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:12.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:12.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:12.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:12.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:12.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:12.898+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:12.668+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:12.668+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:12.668+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:12.669+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:13.434 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:13.137+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6156 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:13.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:13.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:13.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:13.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:13.632+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:13.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:13.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:13.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:13.433+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:13.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:13.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:13.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:13.895+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:14.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:14.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:14.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:14.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:14.606+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:14.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:14.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:14.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:14.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:14.894+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:15.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:15.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:15.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:15.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:15.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:15.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:15.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:15.566+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:15.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:15.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:15.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:15.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:15.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:15.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:15.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:15.911+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:16.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:16.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:16.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:16.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:16.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:16.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:16.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:16.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:16.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:16.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:16.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:16.869+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:17.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:17.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:17.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:17.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:17.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:17.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:17.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:17.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:18.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:17.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:18.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:17.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:18.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:17.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:18.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:17.865+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:18.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:18.139+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6249 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:18.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:18.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:18.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:18.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:18.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:18.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:18.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:18.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:18.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:18.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:18.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:18.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:18.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:18.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:18.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:18.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:18.874+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:18.874+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:18.874+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:18.875+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:19.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:19.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:19.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:19.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:19.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:19.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:19.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:19.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:19.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:19.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:19.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:19.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:20.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:20.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:20.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:20.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:20.309+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:20.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:20.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:20.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:20.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:20.523+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:20.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:20.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:20.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:20.877+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:21.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:21.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:21.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:21.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:21.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:21.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:21.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:21.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:21.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:21.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:21.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:21.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:21.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:21.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:21.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:21.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:21.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:21.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:21.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:21.835+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:22.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:22.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:22.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:22.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:22.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:22.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:22.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:22.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:23.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:22.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:23.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:22.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:23.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:22.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:23.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:22.856+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:23.142 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:23.140+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6342 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:23.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:23.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:23.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:23.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:23.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:23.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:23.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:23.378+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:23.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:23.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:23.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:23.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:23.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:23.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:23.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:23.869+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:24.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:24.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:24.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:24.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:24.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:24.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:24.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:24.415+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:24.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:24.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:24.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:24.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:25.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:25.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:25.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:25.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:25.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:25.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:25.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:25.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:25.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:25.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:25.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:25.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:25.916+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:26.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:26.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:26.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:26.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:26.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:26.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:26.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:26.364+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:26.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:26.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:26.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:27.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:26.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:27.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:27.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:27.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:27.399+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:27.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:27.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:27.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:27.402+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:28.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:27.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:28.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:27.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:28.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:27.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:28.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:27.883+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:28.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:28.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:28.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:28.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:28.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:28.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:28.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:28.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:28.422 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:28.141+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6437 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:28.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:28.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:28.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:28.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:28.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:28.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:28.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:28.909+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:29.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:29.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:29.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:29.453+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:29.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:29.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:29.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:29.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:29.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:29.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:29.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:29.905+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:30.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:30.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:30.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:30.479+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:30.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:30.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:30.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:30.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:30.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:30.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:30.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:30.918+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:31.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:31.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:31.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:31.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:31.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:31.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:31.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:31.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:31.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:31.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:31.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:31.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:32.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:31.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:32.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:31.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:32.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:31.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:31.942+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:32.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:32.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:32.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:32.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:32.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:32.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:32.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:32.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:33.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:33.142+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6531 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:32.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:32.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:32.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:32.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:33.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:33.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:33.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:33.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:33.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:33.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:33.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:33.354+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:33.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:33.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:33.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:33.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:33.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:33.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:33.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:33.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:33.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:33.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:33.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:33.926+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:34.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:34.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:34.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:34.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:34.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:34.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:34.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:34.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:34.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:34.351+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:35.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:34.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:35.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:34.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:35.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:34.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:35.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:34.899+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:35.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:35.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:35.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:35.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:35.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:35.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:35.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:35.335+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:35.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:35.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:35.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:35.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:35.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:35.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:35.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:35.420+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:35.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:35.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:35.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:35.906+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:36.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:36.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:36.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:36.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:36.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:36.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:36.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:36.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:36.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:36.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:36.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:36.870+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:37.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:37.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:37.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:37.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:37.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:37.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:37.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:37.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:38.159 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:38.143+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6625 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:38.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:37.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:38.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:37.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:38.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:37.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:38.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:37.896+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:38.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:38.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:38.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:38.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:38.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:38.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:38.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:38.251+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:38.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:38.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:38.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:38.408+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:38.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:38.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:38.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:39.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:38.898+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:39.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:39.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:39.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:39.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:39.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:39.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:39.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:39.424+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:39.942+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:39.942+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:39.942+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:39.943+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:40.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:40.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:40.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:40.288+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:40.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:40.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:40.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:40.419+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:40.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:40.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:40.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:41.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:40.928+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:41.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:41.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:41.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:41.251+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:41.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:41.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:41.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:41.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:41.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:41.446+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:41.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:41.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:41.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:41.916+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:42.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:42.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:42.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:42.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:42.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:42.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:42.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:42.486+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:43.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:42.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:43.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:42.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:43.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:42.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:43.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:42.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:43.144+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6718 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:43.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:43.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:43.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:43.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:43.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:43.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:43.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:43.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:43.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:43.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:43.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:43.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:43.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:43.874 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:43.524+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:43.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:43.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:43.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:44.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:43.873+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:44.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:44.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:44.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:44.323+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:44.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:44.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:44.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:44.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:44.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:44.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:44.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:44.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:45.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:45.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:45.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:45.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:45.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:45.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:45.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:45.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:45.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:45.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:45.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:45.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:45.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:45.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:45.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:45.588+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:45.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:45.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:45.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:45.888+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:46.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:46.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:46.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:46.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:46.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:46.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:46.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:46.352+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:46.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:46.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:46.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:46.632+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:46.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:46.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:46.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:46.878+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:47.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:47.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:47.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:47.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:47.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:47.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:47.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:47.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:47.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:47.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:19.360990+0000 front 2024-02-20T07:59:19.361197+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:47.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:47.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:19.361016+0000 front 2024-02-20T07:59:19.361089+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:47.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:47.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:19.361128+0000 front 2024-02-20T07:59:19.361136+0000 (oldest deadline 2024-02-20T07:59:45.260995+0000) 2024-02-20T08:09:47.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:47.911+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:47.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:47.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:47.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:47.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:47.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:48.145+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6813 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:48.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:48.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:48.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:48.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:48.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:48.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:48.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:48.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:48.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:48.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:48.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:48.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:48.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:48.953+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:49.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:49.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:49.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:49.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:49.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:49.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:49.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:49.338+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:49.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:49.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:49.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:49.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:49.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:49.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:50.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:49.956+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:50.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:50.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:22.902893+0000 front 2024-02-20T07:59:22.902862+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:50.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:50.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:22.902850+0000 front 2024-02-20T07:59:22.902902+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:50.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:50.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:22.902875+0000 front 2024-02-20T07:59:22.902927+0000 (oldest deadline 2024-02-20T07:59:48.202875+0000) 2024-02-20T08:09:50.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:50.336+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:50.944+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:51.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:50.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:51.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:50.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:50.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:50.727+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:51.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:51.352+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:51.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:51.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:51.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:51.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:51.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:51.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:51.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:51.718+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:51.955+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:52.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:53.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:52.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:53.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:52.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 since back 2024-02-20T07:59:23.267041+0000 front 2024-02-20T07:59:23.267125+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:53.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:52.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-02-20T07:59:23.267095+0000 front 2024-02-20T07:59:23.267072+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:53.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:52.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 since back 2024-02-20T07:59:23.267048+0000 front 2024-02-20T07:59:23.267060+0000 (oldest deadline 2024-02-20T07:59:44.967111+0000) 2024-02-20T08:09:53.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:52.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:53.146+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6907 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:53.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:53.350+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:53.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:53.709+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:53.951+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:54.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:54.745+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:54.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:55.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:55.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:56.024 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:55.729+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:56.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:56.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:56.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:56.326+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:56.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:56.710+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:57.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:57.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:57.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:58.060 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:57.745+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:09:58.147+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7001 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:09:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:58.274+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:58.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:59.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:58.742+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:59.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:09:59.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:59.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:09:59.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:09:59.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T08:09:59.811+0000 7fdfe8693700 0 monclient(hunting): authenticate timed out after 300 2024-02-20T08:09:59.813 INFO:teuthology.orchestra.run.smithi055.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-20T08:09:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:09:59.719+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:00.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:00.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:00.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-20T08:10:00.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:00.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:00.825 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/mon.smithi055/config 2024-02-20T08:10:00.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:00.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:01.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:01.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:01.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:01.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:01.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:02.076+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:02.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:02.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:02.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:02.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:03.382 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:03.147+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7094 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:03.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:03.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:03.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:03.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:03.690+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:04.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:04.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:04.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:04.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:04.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:05.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:05.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:05.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:05.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:05.672+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:06.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:06.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:06.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:06.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:06.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:07.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:07.057+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:07.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:07.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:07.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:07.672+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:08.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:08.149+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7189 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:08.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:08.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:08.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:08.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:08.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:09.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:08.992+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:09.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:09.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:09.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:09.641+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:10.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:09.970+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:10.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:10.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:10.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:10.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:10.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:10.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:10.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:11.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:10.942+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:11.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:11.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:11.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:11.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:11.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:11.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:11.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:11.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:12.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:12.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:12.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:12.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:12.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:12.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:12.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:12.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:12.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:12.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:13.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:13.150+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7283 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:13.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:12.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:13.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:12.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:12.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:12.956+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:13.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:13.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:13.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:13.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:13.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:13.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:13.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:13.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:13.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:13.666+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:13.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:13.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:13.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:14.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:13.942+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:14.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:14.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:14.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:14.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:14.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:14.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:14.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:14.268+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:14.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:14.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:14.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:14.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:14.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:15.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:14.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:15.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:14.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:15.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:14.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:15.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:14.993+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:15.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:15.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:15.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:15.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:15.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:15.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:15.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:15.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:15.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:15.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:15.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:15.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:15.661+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:16.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:16.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:16.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:16.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:16.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:16.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:16.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:16.004+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:16.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:16.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:16.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:16.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:16.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:16.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:16.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:16.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:16.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:16.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:16.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:16.710+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:17.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:17.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:17.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:17.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:17.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:17.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:17.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:17.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:17.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:17.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:17.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:17.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:17.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:17.326+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:17.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:17.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:17.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:17.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:17.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:17.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:17.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:17.711+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:18.356 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:18.151+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7377 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:18.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:18.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:18.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:18.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:18.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:18.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:18.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:18.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:18.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:18.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:18.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:18.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:18.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:18.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:18.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:18.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:18.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:18.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:18.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:18.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:19.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:19.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:19.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:19.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:19.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:19.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:19.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:19.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:19.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:19.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:19.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:19.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:19.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:19.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:19.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:19.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:19.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:19.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:19.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:19.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:19.676+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:20.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:20.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:20.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:20.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:20.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:20.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:20.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:20.071+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:20.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:20.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:20.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:20.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:20.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:20.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:20.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:20.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:20.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:20.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:20.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:20.444+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:21.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:21.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:21.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:21.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:21.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:21.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:21.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:21.096+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:21.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:21.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:21.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:21.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:21.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:21.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:21.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:21.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:21.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:21.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:21.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:21.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:22.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:22.075+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:22.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:22.075+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:22.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:22.075+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:22.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:22.076+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:22.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:22.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:22.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:22.400+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:22.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:22.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:22.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:22.717+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:23.423 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:23.152+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7470 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:23.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:23.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:23.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:23.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:23.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:23.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:23.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:23.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:23.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:23.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:23.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:23.423+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:24.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:23.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:24.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:23.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:24.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:23.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:24.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:23.729+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:24.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:24.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:24.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:24.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:24.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:24.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:24.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:24.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:24.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:25.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:24.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:25.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:24.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:25.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:24.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:25.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:24.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:25.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:25.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:25.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:25.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:25.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:25.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:25.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:25.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:25.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:25.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:25.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:25.475+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:25.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:25.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:25.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:25.772+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:26.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:26.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:26.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:26.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:26.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:26.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:26.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:26.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:26.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:26.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:26.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:26.525+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:27.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:26.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:27.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:26.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:27.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:26.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:27.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:26.781+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:27.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:27.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:27.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:27.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:27.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:27.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:27.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:27.574+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:28.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:27.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:28.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:27.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:28.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:27.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:28.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:27.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:28.153+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7565 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:28.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:28.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:28.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:28.195+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:28.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:28.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:28.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:28.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:28.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:28.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:28.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:28.836+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:28.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:28.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:28.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:28.610+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:29.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:29.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:29.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:29.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:29.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:29.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:29.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:29.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:29.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:29.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:29.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:29.640+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:29.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:29.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:29.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:29.881+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:30.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:30.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:30.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:30.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:30.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:30.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:30.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:30.238+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:30.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:30.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:30.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:30.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:30.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:30.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:30.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:30.654+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:31.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:30.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:31.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:30.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:31.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:30.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:31.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:30.916+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:31.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:31.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:31.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:31.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:31.218+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:31.871 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:31.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:31.871 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:31.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:31.871 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:31.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:31.871 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:31.665+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:31.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:31.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:31.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:31.871+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:32.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:32.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:32.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:32.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:32.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:32.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:32.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:32.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:32.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:32.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:32.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:32.683+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:32.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:32.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:32.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:32.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:33.153+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7659 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:33.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:33.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:33.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:33.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:33.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:33.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:33.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:33.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:33.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:33.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:33.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:33.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:33.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:33.648+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:34.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:33.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:34.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:33.825+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:34.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:34.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:34.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:34.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:34.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:34.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:34.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:34.317+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:34.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:34.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:34.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:34.691+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:34.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:34.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:34.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:34.810+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:35.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:35.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:35.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:35.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:35.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:35.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:35.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:35.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:35.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:35.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:35.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:35.682+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:35.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:35.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:35.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:35.831+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:36.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:36.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:36.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:36.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:36.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:36.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:36.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:36.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:36.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:36.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:36.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:36.799+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:36.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:36.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:36.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:36.655+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:37.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:37.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:37.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:37.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:37.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:37.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:37.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:37.303+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:37.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:37.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:37.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:37.788+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:37.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:37.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:37.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:37.696+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:38.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:38.155+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7753 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:38.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:38.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:38.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:38.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:38.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:38.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:38.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:38.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:38.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:38.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:38.787+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:38.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:38.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:38.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:38.701+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:39.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:39.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:39.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:39.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:39.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:39.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:39.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:39.302+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:39.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:39.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:39.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:39.692+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:39.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:39.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:39.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:39.808+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:40.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:40.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:40.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:40.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:40.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:40.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:40.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:40.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:40.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:40.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:40.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:40.819+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:41.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:41.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:41.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:41.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:41.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:41.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:41.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:41.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:41.737+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:41.737+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:41.737+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:41.738+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:41.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:41.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:41.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:41.780+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:42.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:42.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:42.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:42.278+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:42.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:42.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:42.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:42.785+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:42.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:42.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:42.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:43.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:42.810+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:43.156+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7846 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:43.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:43.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:43.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:43.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:43.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:43.277+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:43.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:43.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:43.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:43.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:43.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:43.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:43.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:43.785+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:44.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:44.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:44.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:44.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:44.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:44.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:44.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:44.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:44.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:44.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:44.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:44.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:45.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:45.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:45.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:45.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:45.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:45.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:45.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:45.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:45.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:45.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:45.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:45.709+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:45.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:45.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:45.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:45.758+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:46.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:46.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:46.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:46.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:46.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:46.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:46.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:46.238+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:46.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:46.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:46.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:46.711+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:46.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:46.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:46.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:46.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:46.723+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:47.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:47.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:47.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:47.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:47.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:47.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:47.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:47.750+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:47.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:47.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:47.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:47.760+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:48.157+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7941 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:48.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:48.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:48.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:48.263+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:48.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:48.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:48.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:48.773+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:48.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:48.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:48.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:48.723+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:49.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:49.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:49.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:49.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:49.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:49.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:49.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:49.748+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:49.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:49.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:49.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:49.746+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:50.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:50.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:50.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:50.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:50.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:50.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:50.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:50.258+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:50.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:50.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:50.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:50.727+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:50.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:50.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:50.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:50.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:51.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:51.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:51.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:51.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:51.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:51.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:51.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:51.232+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:51.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:51.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:51.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:51.679+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:51.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:51.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:51.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:51.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:51.680+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:52.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:52.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:52.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:52.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:52.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:52.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:52.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:52.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:52.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:52.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:52.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:52.675+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:53.156+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8035 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:53.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:53.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:53.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:53.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:53.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:53.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:53.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:53.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:53.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:53.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:53.700+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:53.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:53.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:53.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:53.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:54.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:54.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:54.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:54.150+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:54.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:54.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:54.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:54.683+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:54.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:54.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:54.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:54.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:55.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:55.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:55.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:55.130+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:55.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:55.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:55.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:55.706+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:55.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:55.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:55.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:55.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:56.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:56.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:56.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:56.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:56.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:56.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:56.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:56.744+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:56.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:56.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:56.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:56.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:57.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:57.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:57.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:57.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:57.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:57.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:57.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:57.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:57.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:57.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:57.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:57.749+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:10:58.158+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8129 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:10:58.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:58.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:58.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:58.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:58.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:58.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:58.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:58.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:58.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:58.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:58.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:58.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:58.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:58.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:58.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:58.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:58.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:58.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:58.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:58.703+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:59.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:59.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:59.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:59.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:59.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:59.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:10:59.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:10:59.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:59.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:59.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:59.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:10:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:10:59.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:10:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:59.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:59.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:59.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:10:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:10:59.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.007+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:00.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:00.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:00.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:00.714+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:00.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:00.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:00.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:00.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:00.620+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:00.965+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:01.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:01.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:01.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:01.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:01.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:01.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:01.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:01.745+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:01.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:01.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:01.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:01.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:01.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:01.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:01.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:01.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:01.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:01.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:01.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:01.923+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:02.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:02.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:02.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:02.701+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:02.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:02.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:02.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:02.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:02.729+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:03.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:03.159+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8222 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:03.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:02.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:03.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:02.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:02.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:02.956+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:03.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:03.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:03.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:03.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:03.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:03.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:03.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:03.698+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:03.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:03.709+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:03.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:03.709+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:03.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:03.709+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:03.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:03.710+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:03.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:03.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:03.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:03.933+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:04.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:04.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:04.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:04.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:04.665+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:04.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:04.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:04.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:04.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:04.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:04.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:04.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:04.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:04.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:05.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:05.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:05.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:05.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:05.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:05.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:05.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:05.684+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:05.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:05.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:05.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:05.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:05.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:05.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:05.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:05.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:05.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:05.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:05.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:06.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:05.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:06.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:06.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:06.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:06.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:06.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:06.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:06.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:06.674+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:06.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:06.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:06.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:06.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:06.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:06.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:06.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:06.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:06.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:06.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:06.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:06.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:07.633+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:07.633+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:07.633+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:07.634+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:07.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:07.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:07.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:07.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:07.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:08.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:08.160+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8317 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:08.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:07.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:07.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:07.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:07.961+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:08.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:08.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:08.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:08.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:08.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:08.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:08.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:08.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:08.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:08.654+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:08.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:08.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:08.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:08.954+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:09.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:09.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:09.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:09.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:09.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:09.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:09.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:09.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:09.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:09.563+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:09.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:09.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:09.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:10.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:09.963+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:10.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:10.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:10.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:10.737+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:10.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:10.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:10.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:10.582+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:10.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:10.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:10.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:10.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:11.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:11.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:11.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:11.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:11.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:11.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:11.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:11.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:11.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:11.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:11.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:11.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:11.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:11.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:11.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:11.561+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:11.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:11.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:11.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:11.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:12.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:12.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:12.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:12.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:12.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:12.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:12.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:12.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:12.533+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:13.186 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:13.161+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8411 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:13.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:12.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:13.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:12.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:13.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:12.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:13.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:12.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:13.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:13.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:13.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:13.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:13.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:13.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:13.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:13.563+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:13.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:13.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:13.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:13.794+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:13.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:13.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:13.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:13.925+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:14.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:14.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:14.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:14.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:14.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:14.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:14.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:14.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:14.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:14.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:14.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:14.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:14.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:14.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:14.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:14.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:14.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:14.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:14.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:14.919+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:15.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:15.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:15.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:15.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:15.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:15.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:15.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:15.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:15.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:15.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:15.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:15.805+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:15.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:15.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:15.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:15.930+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:16.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:16.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:16.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:16.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:16.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:16.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:16.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:16.532+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:16.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:16.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:16.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:16.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:16.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:16.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:16.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:16.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:17.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:17.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:17.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:17.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:17.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:17.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:17.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:17.558+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:17.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:17.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:17.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:17.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:17.815+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:17.815+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:17.815+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:18.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:17.816+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:18.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:18.163+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8505 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:18.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:18.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:18.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:18.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:18.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:18.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:18.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:18.782+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:18.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:18.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:18.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:18.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:18.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:18.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:18.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:18.599+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:18.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:18.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:18.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:18.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:19.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:19.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:19.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:19.806+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:19.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:19.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:19.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:19.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:19.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:19.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:19.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:19.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:19.912+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:20.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:20.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:20.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:20.807+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:20.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:20.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:20.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:20.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:20.594+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:20.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:20.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:20.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:20.942+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:21.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:21.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:21.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:21.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:21.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:21.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:21.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:21.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:21.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:21.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:21.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:21.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:21.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:21.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:22.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:22.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:22.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:22.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:22.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:22.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:22.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:22.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:22.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:22.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:22.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:22.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:22.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:22.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:22.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:22.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:23.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:23.164+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8598 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:23.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:22.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:23.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:22.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:23.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:22.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:23.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:22.949+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:23.835 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:23.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:23.835 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:23.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:23.835 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:23.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:23.835 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:23.732+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:23.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:23.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:23.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:23.835+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:23.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:23.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:23.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:23.930+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:24.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:24.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:24.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:24.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:24.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:24.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:24.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:24.884+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:24.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:24.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:24.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:24.764+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:25.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:25.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:25.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:25.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:25.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:25.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:25.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:25.899+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:25.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:25.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:25.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:25.755+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:26.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:26.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:26.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:26.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:26.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:26.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:26.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:26.913+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:26.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:26.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:26.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:26.895+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:27.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:27.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:27.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:27.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:27.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:27.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:28.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:27.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:28.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:27.909+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:28.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:27.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:28.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:27.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:28.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:27.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:28.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:27.779+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:28.165+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8693 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:28.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:28.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:29.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:28.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:28.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:28.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:28.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:28.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:28.913+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:28.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:28.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:28.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:28.906+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:29.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:29.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:29.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:30.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:29.912+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:29.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:29.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:29.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:29.948+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:29.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:29.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:29.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:29.798+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:30.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:30.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:30.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:30.906+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:30.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:30.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:30.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:30.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:31.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:30.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:30.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:30.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:30.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:31.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:31.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:31.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:31.878+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:31.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:31.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:31.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:31.750+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:33.167 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:32.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:33.167 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:32.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:33.167 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:32.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:33.167 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:32.918+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:33.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:33.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:33.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:33.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:32.977+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:33.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:32.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:33.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:32.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:33.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:32.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:33.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:32.770+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:33.166+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8787 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:33.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:33.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:33.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:33.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:33.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:33.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:33.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:33.781+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:33.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:33.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:33.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:33.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:33.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:33.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:33.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:33.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:34.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:34.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:34.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:34.965+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:34.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:34.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:34.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:34.796+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:34.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:34.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:34.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:34.902+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:35.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:35.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:35.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:35.967+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:35.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:35.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:35.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:35.782+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:35.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:35.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:35.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:35.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:37.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:36.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:37.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:36.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:37.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:36.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:37.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:36.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:37.007 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:36.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:37.007 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:36.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:37.008 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:36.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:37.008 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:36.740+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.006+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:37.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:37.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:37.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:37.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:37.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:37.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:37.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:37.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:38.178 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:38.167+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8881 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:38.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:37.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:38.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:37.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:38.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:37.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:38.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:37.923+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:38.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:38.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:38.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:38.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:37.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:38.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:38.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:38.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:38.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:38.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:38.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:38.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:38.896+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:38.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:38.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:38.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:38.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:39.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:39.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:39.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:39.006+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:39.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:39.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:39.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:39.863+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:39.683+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:39.683+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:39.683+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:40.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:39.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:40.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:40.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:40.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:40.030+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:40.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:40.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:40.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:40.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:40.719+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:40.719+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:40.719+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:40.719+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:41.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:41.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:41.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:41.058+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:41.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:41.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:41.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:41.762+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:41.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:41.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:41.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:41.893+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:42.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:42.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:42.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:42.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:42.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:42.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:42.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:42.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:43.073 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:42.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:43.073 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:42.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:43.073 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:42.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:43.073 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:42.878+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:43.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:42.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:43.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:42.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:43.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:42.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:43.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:42.733+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:43.168+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8973 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:43.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:43.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:43.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:43.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:43.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:43.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:43.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:43.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:43.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:43.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:43.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:43.760+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:43.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:43.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:43.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:43.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:44.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:44.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:44.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:44.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:44.064+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:44.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:44.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:44.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:44.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:44.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:44.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:44.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:45.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:44.756+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:45.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:45.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:45.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:45.075+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:45.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:45.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:45.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:45.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:45.744+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:45.744+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:45.744+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:46.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:45.745+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:46.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:46.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:46.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:46.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:47.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:46.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:47.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:46.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:47.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:46.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:47.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:46.919+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:47.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:46.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:47.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:46.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:47.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:46.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:47.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:46.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:47.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:47.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:47.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:47.021+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:48.016 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:47.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:47.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:47.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:47.758+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:47.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:47.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:47.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:48.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:47.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:48.294 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:48.169+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9069 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:48.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:48.016+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:48.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:48.016+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:48.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:48.016+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:48.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:48.017+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:49.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:48.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:49.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:48.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:49.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:48.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:49.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:48.893+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:49.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:48.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:49.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:48.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:49.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:48.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:49.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:48.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:49.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:49.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:49.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:49.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:50.018 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:49.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:50.018 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:49.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:50.018 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:49.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:50.018 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:49.941+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:50.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:49.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:50.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:49.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:50.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:49.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:50.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:49.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:50.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:50.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:50.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:50.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:50.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:50.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:50.335 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:50.019+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:51.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:50.898+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:51.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:50.898+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:51.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:50.898+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:51.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:50.898+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:51.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:50.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:51.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:50.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:51.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:50.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:51.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:50.798+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:51.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:51.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:51.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:51.048+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:51.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:51.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:51.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:51.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:51.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:51.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:51.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:51.819+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.027+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:53.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:52.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:52.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:52.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:52.832+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:53.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:52.985+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:53.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:52.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:53.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:52.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:53.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:52.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:53.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:52.798+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:53.170+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9155 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:53.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:53.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:53.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:53.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:53.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:53.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:53.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:53.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:53.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:53.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:53.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:53.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:53.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:53.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:53.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:53.871+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:54.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:54.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:54.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:54.763+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:54.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:54.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:54.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:54.893+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:54.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:54.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:54.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:54.960+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:55.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:55.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:55.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:55.887+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:55.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:55.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:55.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:55.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:55.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:55.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:55.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:55.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:56.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:56.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:56.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:56.965+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:56.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:56.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:56.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:56.831+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:56.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:56.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:56.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:56.845+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:57.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:57.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:57.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:57.869+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:57.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:57.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:57.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:58.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:57.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:58.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:57.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:58.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:57.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:58.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:57.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:58.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:57.803+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:11:58.171+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9239 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:11:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:58.810+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:58.810+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:58.810+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:11:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:58.811+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:58.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:58.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:58.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:58.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:59.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:59.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:59.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:11:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:11:59.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:00.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:59.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:59.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:59.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:11:59.917+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:00.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:00.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:00.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:00.027+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:59.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:59.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:59.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:11:59.809+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:01.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:00.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:00.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:00.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:00.949+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:01.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:01.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:01.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:01.039+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:00.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:00.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:00.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:00.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:01.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:01.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:01.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:01.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:01.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:01.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:01.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:01.857+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:01.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:01.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:02.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:01.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:01.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:02.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:02.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:02.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:02.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:03.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:02.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:03.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:02.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:03.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:02.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:03.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:02.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:03.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:03.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:03.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:03.105+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:02.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:02.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:02.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:03.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:02.885+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:03.173+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9322 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:04.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:03.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:04.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:03.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:04.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:03.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:04.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:03.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:03.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:03.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:03.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:03.901+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:04.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:04.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:04.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:04.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:05.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:05.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:05.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:05.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:04.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:04.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:04.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:04.967+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:04.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:04.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:04.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:04.901+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:05.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:05.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:05.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:05.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:05.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:05.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:05.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:05.913+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:06.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:06.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:06.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:06.182+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:06.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:06.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:06.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:06.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:07.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:06.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:07.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:06.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:07.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:06.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:07.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:06.869+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:07.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:07.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:07.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:07.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.009+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:07.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:07.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:07.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:07.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:08.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:07.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:08.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:07.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:08.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:07.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:08.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:07.891+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:08.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:08.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:08.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:08.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:08.126+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:08.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:07.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:08.174+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9409 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:08.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:08.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:08.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:09.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:08.905+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:09.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:09.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:09.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:09.079+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:09.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:09.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:09.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:09.001+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:10.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:10.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:10.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:10.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:10.040+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:09.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:09.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:09.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:09.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:10.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:10.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:10.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:10.097+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:11.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:11.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:11.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:11.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:11.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:11.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:11.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:11.086+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:10.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:10.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:10.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:10.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:11.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:11.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:11.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:11.953+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:12.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:12.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:12.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:12.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:12.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:12.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:12.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:12.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:13.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:13.175+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9499 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:13.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:12.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:13.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:12.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:13.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:12.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:13.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:12.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:13.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:13.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:13.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:13.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:13.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:13.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:13.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:13.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:13.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:14.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:14.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:14.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:14.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:13.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:13.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:13.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:13.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:14.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:14.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:14.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:14.140+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:15.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:15.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:15.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:15.028+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:15.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:15.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:15.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:15.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:15.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:15.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:15.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:15.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:16.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:16.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:16.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:16.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:16.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:16.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:16.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:16.097+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:16.993+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:17.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:17.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:17.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:17.072+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:17.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:17.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:17.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:17.138+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:18.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:18.177+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9594 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:18.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:18.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:18.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:18.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:18.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:18.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:18.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:18.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:18.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:18.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:18.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:18.130+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:18.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:18.306 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:18.110+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:19.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:19.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:19.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:19.154+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:19.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:19.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:19.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:19.050+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:19.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:19.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:19.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:19.180+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:20.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:20.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:20.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:20.028+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:20.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:20.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:20.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:20.167+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:20.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:20.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:20.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:20.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:20.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:20.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:20.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:21.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:21.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:21.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:21.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:21.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:21.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:21.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:21.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:21.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:21.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:21.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:21.201+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:22.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:22.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:22.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:22.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:22.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:22.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:22.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:22.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:22.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:22.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:22.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:22.203+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:23.302 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:23.178+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9686 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:23.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:23.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:23.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:23.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:23.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:23.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:23.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:23.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:23.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:23.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:23.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:23.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:23.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:23.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:23.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:23.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:23.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:23.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:24.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:24.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:24.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:24.140+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:24.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:24.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:24.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:24.057+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:24.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:24.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:24.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:24.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.018+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:25.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:25.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:25.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:25.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:25.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:25.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:25.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:25.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:25.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:25.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:26.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:26.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:26.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:26.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:26.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:26.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:26.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:26.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:26.224+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:27.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:27.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:27.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:27.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:27.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:27.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:27.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:27.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:27.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:27.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:27.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:27.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:27.266+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:28.308 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:28.179+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9781 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:28.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:28.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:28.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:28.050+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:28.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:28.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:28.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:28.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:28.089+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:28.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:28.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:28.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:28.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:28.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:28.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:28.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:28.223+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:29.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:29.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:29.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:29.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:29.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:29.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:29.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:29.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:29.274 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:29.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:29.275 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:29.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:29.275 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:29.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:29.275 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:29.099+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:29.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:29.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:29.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:29.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:29.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:29.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:29.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:30.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:30.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:30.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:30.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:30.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:30.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:30.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:30.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:30.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:30.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:30.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:30.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:31.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:31.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:31.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:31.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:31.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:31.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:31.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:31.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:31.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:31.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:31.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:31.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:32.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:32.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:32.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:32.073+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:32.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:32.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:32.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:32.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:32.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:32.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:32.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:32.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:33.342 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:33.180+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9874 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:33.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:33.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:33.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:33.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:33.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:33.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:33.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:33.194+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:33.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:33.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:33.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:33.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:33.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:33.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:33.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:33.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:33.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:33.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:34.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:34.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:34.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:34.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:34.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:34.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:34.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:34.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:34.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:34.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:34.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:34.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:34.268+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:35.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:35.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:35.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:35.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:35.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:35.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:35.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:35.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:35.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:35.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:35.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:35.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:36.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:36.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:36.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:36.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:36.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:36.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:36.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:36.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:36.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:36.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:36.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:36.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:36.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:36.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:36.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:36.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:36.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:36.201+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:37.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:37.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:37.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:37.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:37.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:37.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:37.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:37.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:37.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:37.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:37.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:37.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:38.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:38.181+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9968 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:38.306 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:38.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:38.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:38.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:38.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:38.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:38.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:38.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:38.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:38.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:38.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:38.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:38.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:38.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.034+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:38.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:39.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:39.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:39.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:39.094+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:39.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:39.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:39.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:39.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:39.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:39.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:39.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:39.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:40.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:40.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:40.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:40.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:40.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:40.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:40.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:40.026+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:40.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:40.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:40.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:40.119+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:41.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:41.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:41.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:41.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:41.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:41.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:41.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:41.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:41.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:41.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:41.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:41.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:42.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:42.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:42.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:42.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:42.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:42.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:42.010+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:42.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:42.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:42.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:42.084+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:42.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:42.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:42.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:42.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:42.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:43.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:43.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:43.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:43.308 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:43.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:43.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:43.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:43.088+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:43.183+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10060 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:43.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:43.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:43.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:43.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:43.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:43.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:43.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:44.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:44.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:44.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:44.130+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:44.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:44.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:44.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:44.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:45.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:45.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:45.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:45.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:44.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:44.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:44.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:44.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:45.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:45.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:45.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:45.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:46.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:45.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:46.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:45.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:46.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:45.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:46.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:45.922+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:46.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:46.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:46.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:46.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:46.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:46.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:46.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:46.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:46.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:46.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:46.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:46.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:47.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:47.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:47.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:47.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:47.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:47.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:47.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:47.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:47.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:46.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:47.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:46.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:47.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:46.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:47.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:46.877+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:47.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:47.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:47.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:47.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:48.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:47.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:48.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:47.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:48.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:47.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:48.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:47.830+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:48.184+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10155 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:48.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:48.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:48.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:48.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:48.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:48.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:48.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:48.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:48.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:48.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:48.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:48.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:48.819+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:49.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:49.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:49.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:49.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:49.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:49.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:49.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:49.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:50.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:49.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:49.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:49.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:49.853+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:50.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:50.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:50.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:50.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:50.050+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:50.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:50.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:50.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:50.186+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:51.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:50.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:51.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:50.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:51.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:50.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:51.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:50.809+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:51.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:51.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:51.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:51.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:51.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:51.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:51.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:51.092+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:51.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:51.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:51.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:51.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:51.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:51.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:51.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:51.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:51.778+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:52.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:52.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:52.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:52.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:52.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:52.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:52.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:52.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:52.121+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:53.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:52.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:53.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:52.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:53.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:52.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:53.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:52.798+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:53.185+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10249 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:53.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:53.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:53.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:53.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:53.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:53.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:53.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:53.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:54.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:53.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:54.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:53.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:54.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:53.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:54.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:53.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:54.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:54.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:54.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:54.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:54.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:54.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:54.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:54.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:54.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:55.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:54.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:55.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:54.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:55.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:54.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:55.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:54.828+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:55.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:55.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:55.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:55.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:55.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:55.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:55.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:55.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:55.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:55.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:55.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:55.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:56.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:55.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:56.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:55.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:56.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:55.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:56.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:55.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:56.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:56.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:56.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:56.199+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:56.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:56.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:56.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:56.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:57.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:57.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:57.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:57.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:57.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:57.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:57.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:57.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:56.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:56.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:56.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:56.840+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:57.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:57.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:57.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:57.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:58.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:57.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:58.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:57.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:58.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:57.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:58.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:57.829+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:12:58.186+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10344 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:12:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:58.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:58.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:58.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:58.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:58.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:58.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:58.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:58.127+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:58.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:58.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:58.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:12:59.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:58.873+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:59.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:59.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:59.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:12:59.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:59.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:59.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:59.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:12:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:12:59.114+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:00.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:00.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:00.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:00.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:00.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:00.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:00.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:00.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:00.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:59.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:00.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:59.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:00.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:59.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:00.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:12:59.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:00.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:00.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:00.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:00.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:00.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:00.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:00.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:00.895+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:01.062+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:01.062+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:01.062+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:01.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:01.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:01.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:01.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:01.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:01.165+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:01.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:01.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:01.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:01.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:02.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:02.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:02.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:02.105+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:02.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:02.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:02.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:02.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:02.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:02.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:02.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:02.191+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:03.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:02.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:02.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:02.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:02.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:03.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:03.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:03.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:03.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:03.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:03.188+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10436 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:03.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:03.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:03.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:03.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:03.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:04.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:03.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:04.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:03.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:04.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:03.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:04.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:03.894+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:04.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:04.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:04.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:04.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:04.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:04.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:04.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:04.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:04.258+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:05.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:05.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:05.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:05.143+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:04.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:05.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:04.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:04.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:04.908+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:05.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:05.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:05.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:05.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:06.133 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:05.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:06.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:05.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:06.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:05.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:06.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:05.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:06.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:06.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:06.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:06.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:06.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:06.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:06.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:06.267+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:06.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:06.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:07.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:06.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:06.918+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:07.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:07.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:07.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:07.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:07.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:07.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:07.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:07.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:07.302+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:08.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:07.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:08.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:07.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:08.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:07.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:08.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:07.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:08.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:08.114+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:08.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:08.114+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:08.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:08.114+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:08.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:08.115+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:08.189+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10532 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:08.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:08.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:08.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:08.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:08.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:08.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:09.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:08.859+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:09.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:08.859+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:09.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:08.859+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:09.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:08.860+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:09.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:09.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:09.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:09.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:09.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:09.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:09.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:09.351+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:10.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:09.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:10.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:09.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:10.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:09.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:10.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:09.840+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:10.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:10.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:10.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:10.366+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:10.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:10.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:10.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:10.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:10.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:11.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:11.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:11.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:11.088+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:10.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:10.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:10.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:10.839+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:11.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:11.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:11.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:11.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:11.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:11.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:11.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:11.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:11.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:11.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:11.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:11.861+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:12.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:12.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:12.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:12.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:12.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:12.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:12.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:12.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:12.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:12.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:12.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:12.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:12.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:12.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:12.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:12.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:13.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:12.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:13.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:12.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:13.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:12.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:13.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:12.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:13.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:13.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:13.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:13.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:13.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:13.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:13.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:13.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:13.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:13.190+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10625 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:13.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:13.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:13.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:13.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:13.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:13.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:13.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:14.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:13.859+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:14.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:13.859+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:14.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:13.859+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:14.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:13.860+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:14.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:14.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:14.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:14.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:14.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:14.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:14.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:14.126+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:14.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:14.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:14.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:14.433+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:15.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:14.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:15.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:14.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:15.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:14.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:15.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:14.838+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:15.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:15.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:15.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:15.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:15.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:15.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:15.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:15.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:15.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:15.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:15.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:15.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:15.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:15.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:15.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:15.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:16.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:15.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:16.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:15.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:16.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:15.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:16.135 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:15.837+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:16.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:16.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:16.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:16.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:16.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:16.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:16.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:16.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:16.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:16.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:16.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:16.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:16.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:16.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:16.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:16.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:17.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:16.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:17.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:16.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:17.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:16.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:17.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:16.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:17.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:17.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:17.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:17.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:17.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:17.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:17.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:17.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:17.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:17.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:17.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:17.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:17.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:17.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:17.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:17.441+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:18.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:17.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:18.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:17.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:18.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:17.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:18.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:17.798+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:18.453 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:18.192+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10720 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:18.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:18.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:18.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:18.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:18.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:18.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:18.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:18.194+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:18.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:18.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:18.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:18.453+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:18.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:18.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:18.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:19.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:18.793+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:19.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:19.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:19.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:19.244+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:19.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:19.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:19.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:19.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:19.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:19.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:19.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:20.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:19.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:20.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:20.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:20.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:20.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:20.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:20.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:20.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:20.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:20.446+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:20.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:20.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:20.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:21.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:20.785+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:21.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:21.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:21.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:21.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:21.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:21.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:21.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:21.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:21.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:21.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:21.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:21.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:21.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:21.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:21.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:21.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:21.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:21.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:22.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:22.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:22.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:22.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:22.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:22.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:22.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:22.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:22.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:22.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:22.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:22.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:22.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:22.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:22.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:22.514+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:23.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:22.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:23.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:22.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:23.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:22.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:23.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:22.799+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:23.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:23.193+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10812 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:23.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:23.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:23.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:23.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:23.266+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:23.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:23.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:23.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:23.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:23.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:23.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:23.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:23.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:23.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:23.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:23.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:23.840+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:24.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:24.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:24.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:24.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:24.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:24.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:24.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:24.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:24.306+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:24.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:24.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:24.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:24.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:25.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:25.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:25.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:25.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:25.350+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:25.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:25.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:25.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:25.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:25.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:25.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:25.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:25.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:26.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:26.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:26.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:26.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:26.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:26.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:26.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:26.323+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:26.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:26.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:26.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:26.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:26.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:26.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:26.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:26.591+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:26.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:26.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:26.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:26.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:26.834+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:27.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:27.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:27.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:27.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:27.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:27.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:27.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:27.549+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:28.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:27.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:28.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:27.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:28.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:27.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:28.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:27.823+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:28.193+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10906 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:28.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:28.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:28.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:28.319+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:28.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:28.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:28.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:28.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:28.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:28.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:28.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:28.589+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:28.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:28.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:28.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:28.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:29.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:29.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:29.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:29.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:29.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:29.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:29.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:29.600+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:30.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:29.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:30.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:29.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:30.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:29.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:30.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:29.809+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:30.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:30.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:30.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:30.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:30.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:30.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:30.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:30.393+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:30.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:30.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:30.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:30.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:30.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:30.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:30.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:30.766+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:31.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:31.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:31.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:31.565+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:31.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:31.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:31.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:31.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:31.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:31.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:31.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:31.746+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:31.746+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:32.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:32.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:32.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:32.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:32.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:32.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:32.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:32.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:33.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:32.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:33.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:32.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:33.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:32.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:33.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:32.779+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:33.195+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10997 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:33.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:33.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:33.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:33.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:33.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:33.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:33.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:33.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:33.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:33.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:33.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:33.739+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:33.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:33.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:33.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:33.543+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:34.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:34.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:34.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:34.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:34.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:34.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:34.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:34.507+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:34.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:34.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:34.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:34.693+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:34.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:34.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:34.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:34.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:35.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:35.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:35.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:35.537+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:35.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:35.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:35.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:35.645+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:35.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:35.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:35.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:35.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:36.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:36.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:36.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:36.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:36.516+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:36.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:36.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:36.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:36.602+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:36.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:36.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:36.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:36.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:37.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:37.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:37.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:37.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:37.592+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:37.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:37.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:37.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:37.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:37.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:37.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:37.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:37.492+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:38.448 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:38.196+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11091 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:38.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:38.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:38.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:38.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:38.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:38.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:38.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:38.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:38.448+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:38.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:38.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:38.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:38.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:39.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:39.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:39.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:39.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:39.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:39.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:39.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:39.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:39.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:39.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:39.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:39.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:40.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:40.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:40.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:40.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:40.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:40.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:40.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:40.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:40.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:40.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:40.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:40.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:40.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:40.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:40.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:40.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:41.671 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:41.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:41.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:41.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:41.535+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:41.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:41.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:41.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:41.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:41.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:41.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:41.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:41.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:41.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:41.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:41.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:41.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:41.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:42.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:42.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:42.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:42.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:42.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:42.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:42.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:42.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:42.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:42.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:42.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:42.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:42.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:42.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:42.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:42.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:42.552+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:43.198+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11182 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:43.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:43.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:43.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:43.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:43.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:43.345+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:43.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:43.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:43.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:43.265+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:43.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:43.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:43.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:43.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:43.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:43.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:43.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:43.518+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:44.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:44.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:44.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:44.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:44.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:44.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:44.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:44.379+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:44.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:44.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:44.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:44.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:45.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:45.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:45.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:45.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:45.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:45.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:45.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:45.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:45.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:46.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:46.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:46.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:46.444+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:46.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:46.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:46.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:46.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:46.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:46.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:46.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:46.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:47.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:47.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:47.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:47.314+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:47.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:47.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:47.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:47.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:47.315+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:47.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:47.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:47.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:47.413+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:48.199+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11276 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:48.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:48.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:48.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:48.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:48.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:48.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:48.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:48.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:48.370+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:48.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:48.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:48.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:48.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:49.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:49.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:49.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:49.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:49.277+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:49.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:49.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:49.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:49.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:49.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:49.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:49.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:49.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:50.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:50.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:50.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:50.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:50.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:50.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:50.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:50.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:50.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:50.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:50.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:50.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:51.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:51.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:51.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:51.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:51.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:51.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:51.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:51.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:51.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:51.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:51.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:51.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:51.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:51.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:51.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:51.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:51.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:52.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:52.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:52.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:52.219+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:52.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:52.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:52.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:52.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:52.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:52.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:52.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:52.258+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:53.201+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11368 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:53.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:53.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:53.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:53.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:53.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:53.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:53.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:53.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:53.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:53.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:53.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:53.416+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:54.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:54.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:54.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:54.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:54.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:54.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:54.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:54.238+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:54.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:54.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:54.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:54.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:55.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:55.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:55.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:55.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:55.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:55.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:55.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:55.283+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:55.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:55.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:55.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:55.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:56.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:56.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:56.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:56.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:56.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:56.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:56.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:56.389+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:56.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:56.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:56.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:56.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:57.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:57.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:57.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:57.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:57.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:57.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:57.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:57.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:57.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:57.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:57.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:57.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:57.254+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:57.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:57.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:57.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:57.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:57.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:57.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:58.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:13:58.202+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11462 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:13:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:58.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:58.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:58.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:58.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:58.223+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:58.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:58.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:58.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:58.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:58.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:58.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:58.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:58.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:59.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:59.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:59.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:13:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:13:59.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:59.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:59.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:59.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:13:59.405+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:59.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:59.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:59.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:13:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:13:59.271+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:00.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:00.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:00.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:00.254+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:00.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:00.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:00.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:00.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:00.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:01.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:01.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:01.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:01.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:01.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:01.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:01.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:01.461+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:01.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:01.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:01.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:01.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:01.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:02.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:02.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:02.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:02.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:02.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:02.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:02.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:02.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:02.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:02.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:02.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:02.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:02.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:03.203+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11553 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:03.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:03.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:03.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:03.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:03.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:03.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:03.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:03.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:03.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:03.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:03.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:03.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:03.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:03.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:04.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:04.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:04.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:04.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:04.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:04.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:04.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:04.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:04.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:04.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:04.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:04.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:04.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:04.324+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:05.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:05.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:05.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:05.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:05.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:05.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:05.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:05.362+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:05.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:05.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:05.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:05.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:06.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:06.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:06.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:06.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:06.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:06.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:06.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:06.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:06.358+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:06.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:06.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:06.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:06.345+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:07.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:07.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:07.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:07.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:07.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:07.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:07.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:07.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:07.416+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:07.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:07.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:07.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:07.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:08.204+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11648 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:08.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:08.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:08.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:08.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:08.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:08.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:08.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:08.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:08.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:08.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:08.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:08.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:08.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:08.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:08.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:09.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:09.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:09.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:09.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:09.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:09.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:09.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:09.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:09.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:09.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:09.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:09.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:10.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:10.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:10.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:10.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:10.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:10.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:10.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:10.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:10.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:10.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:10.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:10.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:10.461+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:10.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:10.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:10.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:10.428+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:11.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:11.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:11.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:11.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:11.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:11.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:11.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:11.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:11.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:11.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:11.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:11.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:11.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:11.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:11.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:11.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:11.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:11.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:12.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:12.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:12.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:12.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:12.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:12.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:12.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:12.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:12.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:12.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:12.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:12.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:12.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:12.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:12.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:12.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:12.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:12.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:12.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:12.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:13.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:13.206+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11740 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:13.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:13.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:13.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:13.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:13.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:13.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:13.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:13.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:13.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:13.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:13.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:13.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:13.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:13.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:13.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:13.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:14.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:14.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:14.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:14.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:14.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:14.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:14.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:14.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:14.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:14.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:14.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:14.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:15.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:15.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:15.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:15.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:15.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:15.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:15.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:15.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:15.421+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:15.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:15.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:15.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:15.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:16.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:16.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:16.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:16.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:16.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:16.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:16.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:16.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:16.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:16.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:16.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:16.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:16.545+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:17.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:17.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:17.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:17.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:17.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:17.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:17.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:17.366+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:17.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:17.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:17.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:17.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:18.206+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11834 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:18.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:18.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:18.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:18.114+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:18.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:18.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:18.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:18.401+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:18.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:18.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:18.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:18.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:19.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:19.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:19.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:19.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:19.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:19.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:19.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:19.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:19.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:19.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:19.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:19.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:20.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:20.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:20.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:20.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:20.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:20.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:20.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:20.178+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:20.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:20.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:20.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:20.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:20.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:20.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:20.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:20.557+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:21.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:21.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:21.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:21.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:21.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:21.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:21.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:21.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:21.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:21.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:21.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:21.349+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:21.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:21.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:21.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:21.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:22.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:22.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:22.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:22.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:22.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:22.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:22.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:22.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:22.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:22.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:22.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:22.534+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:23.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:23.208+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11925 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:23.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:23.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:23.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:23.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:23.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:23.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:23.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:23.146+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:23.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:23.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:23.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:23.285+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:23.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:23.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:23.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:23.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:23.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:23.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:23.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:23.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:24.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:24.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:24.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:24.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:24.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:24.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:24.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:24.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:24.187+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:24.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:24.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:24.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:24.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:25.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:25.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:25.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:25.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:25.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:25.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:25.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:25.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:25.228+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:25.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:25.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:25.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:25.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:25.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:25.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:25.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:25.559+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:26.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:26.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:26.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:26.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:26.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:26.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:26.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:26.278+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:26.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:26.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:26.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:26.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:26.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:26.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:26.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:26.259+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:26.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:26.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:26.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:26.582+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:27.227+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:27.227+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:27.227+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:27.228+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:27.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:27.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:27.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:27.241+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:27.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:27.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:27.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:27.623+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:28.210+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12020 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:28.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:28.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:28.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:28.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:28.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:28.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:28.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:28.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:28.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:28.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:28.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:28.632+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:29.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:29.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:29.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:29.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:29.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:29.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:29.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:29.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:29.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:29.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:29.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:29.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:29.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:29.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:29.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:29.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:29.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:29.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:29.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:29.610+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:30.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:30.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:30.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:30.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:30.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:30.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:30.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:30.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:30.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:30.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:30.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:30.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:30.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:30.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:30.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:30.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:30.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:30.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:30.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:30.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:31.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:31.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:31.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:31.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:31.302+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:31.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:31.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:31.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:31.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:31.337+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:31.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:31.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:31.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:31.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:32.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:32.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:32.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:32.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:32.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:32.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:32.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:32.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:32.331+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:32.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:32.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:32.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:32.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:33.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:33.211+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12113 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:33.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:33.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:33.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:33.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:33.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:33.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:33.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:33.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:33.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:33.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:33.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:33.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:33.266+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:33.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:33.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:33.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:33.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:34.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:34.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:34.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:34.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:34.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:34.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:34.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:34.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:34.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:34.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:34.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:34.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:34.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:34.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:34.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:34.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:34.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:34.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:34.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:34.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:34.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:35.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:35.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:35.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:35.474+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:35.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:35.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:35.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:35.282+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:35.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:35.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:35.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:35.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:36.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:36.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:36.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:36.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:36.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:36.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:36.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:36.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:36.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:36.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:36.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:36.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:36.268+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:37.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:37.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:37.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:37.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:37.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:37.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:37.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:37.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:37.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:37.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:37.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:37.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:38.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:38.212+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12206 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:38.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:38.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:38.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:38.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:38.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:38.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:38.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:38.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:38.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:38.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:38.337+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:38.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:38.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:38.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:38.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:39.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:39.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:39.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:39.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:39.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:39.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:39.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:39.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:39.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:39.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:39.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:39.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:40.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:40.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:40.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:40.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:40.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:40.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:40.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:40.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:40.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:40.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:40.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:40.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:40.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:40.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:40.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:40.289+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:40.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:40.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:40.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:40.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:40.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:41.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:41.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:41.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:41.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:41.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:41.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:41.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:41.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:41.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:41.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:41.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:41.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:41.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:41.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:41.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:41.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:41.292+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:42.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:42.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:42.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:42.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:42.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:42.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:42.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:42.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:42.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:42.306+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:42.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:42.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:42.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:42.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:43.214+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12298 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:43.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:43.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:43.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:43.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:43.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:43.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:43.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:43.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:43.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:43.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:43.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:43.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:43.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:43.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:43.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:43.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:44.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:44.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:44.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:44.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:44.253+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:44.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:44.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:44.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:44.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:44.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:44.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:44.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:44.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:44.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:45.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:45.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:45.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:45.373+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:45.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:45.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:45.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:45.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:45.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:45.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:45.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:45.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:45.578+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:46.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:46.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:46.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:46.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:46.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:46.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:46.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:46.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:46.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:46.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:46.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:46.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:46.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:47.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:47.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:47.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:47.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:47.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:47.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:47.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:47.279+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:47.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:47.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:47.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:47.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:47.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:47.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:47.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:47.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:47.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:47.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:47.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:47.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:48.215+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12392 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:48.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:48.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:48.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:48.231+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:48.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:48.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:48.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:48.353+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:48.563+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:48.563+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:48.563+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:48.564+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:49.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:49.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:49.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:49.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:49.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:49.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:49.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:49.218+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:49.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:49.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:49.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:49.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:49.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:49.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:49.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:49.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:49.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:49.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:49.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:49.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:50.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:50.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:50.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:50.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:50.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:50.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:50.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:50.307+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:50.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:50.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:50.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:50.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:51.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:51.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:51.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:51.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:51.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:51.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:51.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:51.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:51.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:51.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:51.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:51.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:51.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:52.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:52.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:52.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:52.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:52.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:52.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:52.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:52.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:52.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:52.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:52.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:52.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:52.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:52.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:52.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:52.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:52.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:52.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:52.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:52.607+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:53.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:53.216+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12485 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:53.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:53.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:53.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:53.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:53.281+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:53.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:53.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:53.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:53.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:53.280+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:53.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:53.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:53.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:53.628+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:54.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:54.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:54.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:54.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:54.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:54.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:54.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:54.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:54.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:54.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:54.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:54.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:54.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:54.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:54.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:54.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:54.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:54.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:54.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:54.628+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:55.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:55.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:55.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:55.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:55.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:55.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:55.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:55.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:55.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:55.282+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:55.674+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:55.674+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:55.674+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:55.674+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:56.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:56.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:56.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:56.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:56.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:56.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:56.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:56.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:56.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:56.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:56.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:56.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:56.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:56.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:56.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:56.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:56.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:56.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:56.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:56.703+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:57.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:57.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:57.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:57.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:57.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:57.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:57.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:57.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:57.329+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:57.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:57.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:57.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:57.718+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:58.230 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:14:58.217+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12577 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:14:58.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:58.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:58.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:58.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:58.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:58.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:58.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:58.322+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:58.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:58.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:58.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:58.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:58.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:58.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:58.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:58.345+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:58.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:58.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:58.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:58.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:59.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:59.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:59.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:14:59.312+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:59.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:59.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:59.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:14:59.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:14:59.343+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:14:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:59.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:59.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:59.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:14:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:14:59.674+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:00.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:00.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:00.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:00.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:00.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:00.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:00.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:00.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:00.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:00.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:00.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:00.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:00.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:00.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:00.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:00.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:00.682+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:01.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:01.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:01.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:01.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:01.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:01.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:01.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:01.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:01.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:01.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:01.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:01.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:01.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:01.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:01.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:01.327+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:01.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:01.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:01.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:01.678+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:02.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-02-20T08:15:02.496+0000 7f563f684700 0 monclient(hunting): authenticate timed out after 300 2024-02-20T08:15:02.498 INFO:teuthology.orchestra.run.smithi055.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-20T08:15:02.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:02.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:02.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:02.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:02.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:02.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:02.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:02.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:02.291+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:02.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:02.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:02.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:02.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:02.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:02.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:02.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:02.686+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:03.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:03.219+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12669 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:03.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:03.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:03.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:03.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:03.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:03.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:03.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:03.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:03.246+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T08:15:03.501 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-20T08:15:03.706 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c4295564a6b64dc499a0947839ce008f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-20T08:15:03.710 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-20T08:15:03.722 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-20T08:15:03.723 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T08:15:03.723 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:15:03.754 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T08:15:03.754 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:15:03.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-20T08:15:04.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:03.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:04.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:03.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:04.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:03.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:04.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:03.728+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:04.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:04.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:04.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:04.236+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:04.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:04.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:04.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:04.203+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:05.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:04.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:05.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:04.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:05.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:04.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:05.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:04.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:05.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:05.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:05.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:05.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:05.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:05.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:05.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:05.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:05.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:05.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:05.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:05.285+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:06.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:05.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:06.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:05.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:06.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:05.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:06.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:05.795+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:06.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:06.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:06.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:06.306+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:06.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:06.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:06.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:06.203+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:07.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:06.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:07.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:06.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:07.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:06.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:07.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:06.765+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:07.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:07.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:07.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:07.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:07.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:07.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:07.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:07.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:07.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:07.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:07.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:07.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:07.717+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:08.220+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12764 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:08.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:08.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:08.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:08.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:08.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:08.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:08.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:08.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:08.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:08.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:08.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:08.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:08.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:09.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:09.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:09.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:09.281+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:09.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:09.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:09.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:09.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:10.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:09.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:10.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:09.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:10.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:09.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:10.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:09.736+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:10.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:10.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:10.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:10.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:10.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:10.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:10.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:10.333+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:10.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:10.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:10.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:10.308+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:10.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:10.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:10.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:10.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:11.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:11.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:11.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:11.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:11.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:11.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:11.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:11.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:11.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:11.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:11.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:11.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:11.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:11.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:11.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:11.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:11.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:11.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:11.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:11.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:12.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:12.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:12.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:12.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:12.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:12.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:12.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:12.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:12.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:12.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:12.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:12.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:12.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:12.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:12.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:12.309+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:12.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:12.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:12.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:12.717+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:13.345 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:13.221+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12858 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:13.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:13.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:13.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:13.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:13.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:13.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:13.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:13.282+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:13.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:13.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:13.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:13.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:13.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:13.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:13.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:13.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:13.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:13.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:13.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:13.699+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:14.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:14.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:14.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:14.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:14.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:14.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:14.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:14.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:14.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:14.350+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:14.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:14.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:14.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:14.706+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:15.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:15.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:15.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:15.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:15.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:15.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:15.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:15.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:15.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:15.312+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:15.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:15.312+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:15.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:15.312+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:15.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:15.313+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:15.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:15.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:15.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:15.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:16.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:16.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:16.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:16.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:16.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:16.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:16.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:16.311+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:16.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:16.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:16.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:16.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:17.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:17.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:17.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:17.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:17.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:17.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:17.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:17.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:17.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:17.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:17.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:17.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:17.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:17.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:17.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:17.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:17.694+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:17.694+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:17.694+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:17.695+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:18.223+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12953 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:18.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:18.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:18.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:18.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:18.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:18.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:18.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:18.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:18.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:18.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:18.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:18.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:19.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:19.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:19.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:19.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:19.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:19.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:19.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:19.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:19.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:19.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:19.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:19.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:19.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:19.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:19.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:19.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:19.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:19.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:19.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:19.719+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:20.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:20.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:20.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:20.298+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:20.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:20.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:20.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:20.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:20.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:20.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:20.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:20.694+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:21.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:21.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:21.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:21.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:21.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:21.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:21.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:21.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:21.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:21.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:21.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:21.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:21.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:21.652+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:22.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:22.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:22.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:22.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:22.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:22.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:22.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:22.318+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:22.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:22.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:22.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:22.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:22.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:22.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:22.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:22.236+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:22.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:22.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:22.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:22.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:22.686+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:23.570 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:23.224+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13045 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:23.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:23.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:23.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:23.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:23.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:23.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:23.571 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:23.236+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:23.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:23.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:23.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:23.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:23.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:23.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:23.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:23.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:23.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:23.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:23.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:23.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:23.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:23.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:23.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:23.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:24.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:24.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:24.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:24.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:24.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:24.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:24.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:24.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:24.190+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:24.190+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:24.190+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:24.191+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:24.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:24.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:24.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:24.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:24.681+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:25.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:25.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:25.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:25.222+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:25.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:25.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:25.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:25.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:25.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:25.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:25.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:25.353+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:25.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:25.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:25.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:25.644+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:26.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:26.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:26.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:26.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:26.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:26.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:26.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:26.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:26.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:26.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:26.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:26.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:26.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:26.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:26.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:26.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:26.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:26.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:26.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:26.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:26.686+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:27.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:27.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:27.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:27.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:27.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:27.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:27.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:27.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:27.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:27.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:27.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:27.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:27.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:27.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:27.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:27.371+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:27.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:27.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:27.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:27.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:27.672+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:28.225+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13140 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:28.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:28.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:28.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:28.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:28.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:28.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:28.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:28.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:28.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:28.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:28.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:28.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:28.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:28.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:28.624+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:29.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:29.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:29.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:29.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:29.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:29.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:29.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:29.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:29.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:29.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:29.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:29.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:29.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:29.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:29.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:29.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:29.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:29.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:29.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:29.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:29.588+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:30.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:30.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:30.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:30.297+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:30.371+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:30.371+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:30.371+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:30.372+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:30.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:30.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:30.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:30.626+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:31.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:31.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:31.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:31.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:31.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:31.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:31.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:31.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:31.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:31.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:31.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:31.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:31.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:31.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:31.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:31.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:31.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:31.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:31.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:31.633+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:32.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:32.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:32.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:32.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:32.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:32.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:32.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:32.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:32.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:32.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:32.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:32.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:32.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:32.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:32.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:32.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:32.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:32.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:32.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:32.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:32.654+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:33.571 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:33.227+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13234 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:33.571 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:33.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:33.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:33.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:33.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:33.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:33.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:33.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:33.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:33.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:33.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:33.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:33.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:33.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:33.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:33.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:33.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:33.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:33.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:33.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:33.658+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:34.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:34.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:34.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:34.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:34.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:34.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:34.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:34.624+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:34.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:34.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:34.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:34.408+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:35.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:35.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:35.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:35.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:35.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:35.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:35.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:35.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:35.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:35.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:35.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:35.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:35.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:35.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:35.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:35.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:35.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:35.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:35.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:35.662+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:36.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:36.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:36.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:36.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:36.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:36.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:36.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:36.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:36.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:36.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:36.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:36.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:36.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:36.668+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:36.668+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:36.668+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:36.669+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:37.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:37.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:37.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:37.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:37.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:37.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:37.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:37.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:37.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:37.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:37.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:37.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:37.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:37.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:37.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:37.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:37.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:37.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:37.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:37.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:37.696+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:38.690 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:38.228+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13328 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:38.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:38.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:38.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:38.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:38.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:38.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:38.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:38.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:38.691 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:38.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:38.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:38.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:38.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:38.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:38.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:38.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:38.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:38.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:38.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:38.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:38.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:39.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:39.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:39.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:39.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:39.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:39.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:39.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:39.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:39.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:40.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:39.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:40.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:39.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:40.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:39.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:40.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:39.728+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:40.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:40.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:40.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:40.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:40.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:40.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:40.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:40.392+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:40.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:40.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:40.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:40.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:40.766+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:40.766+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:40.766+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:40.767+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:41.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:41.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:41.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:41.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:41.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:41.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:41.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:41.485+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:41.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:41.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:41.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:41.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:41.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:41.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:41.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:41.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:41.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:41.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:41.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:41.814+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:42.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:42.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:42.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:42.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:42.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:42.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:42.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:42.496+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:42.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:42.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:42.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:42.806+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:43.572 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:43.230+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13421 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:43.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:43.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:43.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:43.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:43.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:43.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:43.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:43.353+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:43.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:43.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:43.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:43.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:43.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:43.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:43.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:43.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:44.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:43.846+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:44.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:43.846+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:43.846+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:43.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:44.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:44.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:44.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:44.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:44.485+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:44.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:44.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:44.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:44.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:44.874+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:44.874+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:44.874+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:45.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:44.875+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:45.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:45.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:45.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:45.532+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:45.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:45.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:45.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:45.424+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:45.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:45.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:45.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:45.908+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:46.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:46.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:46.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:46.466+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:46.527+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:46.527+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:46.527+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:46.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:47.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:46.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:47.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:46.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:47.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:46.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:47.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:46.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:47.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:47.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:47.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:47.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:47.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:47.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:47.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:47.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:47.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:47.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:47.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:47.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:47.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:47.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:47.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:47.479+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:47.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:47.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:47.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:47.930+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:48.520 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:48.231+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13516 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:48.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:48.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:48.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:48.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:48.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:48.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:48.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:48.521+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:48.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:48.526+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:48.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:48.526+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:48.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:48.526+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:48.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:48.527+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:48.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:48.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:48.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:49.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:48.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:49.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:49.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:49.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:49.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:49.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:49.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:49.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:49.554+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:49.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:49.560+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:49.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:49.560+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:49.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:49.560+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:49.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:49.561+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:49.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:49.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:49.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:49.965+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:50.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:50.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:50.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:50.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:50.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:50.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:50.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:50.516+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:50.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:50.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:50.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:50.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:50.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:50.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:50.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:51.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:50.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:51.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:51.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:51.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:51.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:51.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:51.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:51.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:51.599+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.019+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:52.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:52.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:52.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:52.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:52.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:52.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:52.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:52.575+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:52.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:52.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:52.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:52.554+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:53.255 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:53.232+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13610 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:53.255 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:53.255 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:53.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:53.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:52.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:53.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:53.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:53.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:53.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:53.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:53.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:53.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:53.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:53.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:54.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:54.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:54.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:54.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:54.035+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:54.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:54.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:54.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:54.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:54.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:54.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:54.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:54.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:54.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:54.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:54.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:54.559+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:55.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:55.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:55.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:55.084+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:55.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:55.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:55.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:55.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:55.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:55.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:55.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:55.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:55.570+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:56.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:56.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:56.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:56.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:56.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:56.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:56.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:56.594+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:56.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:56.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:56.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:56.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:57.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:57.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:57.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:57.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:57.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:57.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:57.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:57.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:57.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:57.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:57.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:57.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:57.498+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:58.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:15:58.234+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13704 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:15:58.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:58.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:58.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:58.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:58.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:58.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:58.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:58.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:58.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:58.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:58.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:58.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:58.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:58.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:58.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:58.470+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:58.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:58.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:58.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:58.601+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:59.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:59.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:59.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:59.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:59.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:59.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:15:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:15:59.041+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:59.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:59.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:59.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:15:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:15:59.514+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:15:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:59.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:59.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:59.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:15:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:15:59.641+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:00.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:00.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:00.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:00.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:00.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:00.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:00.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:00.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:00.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:00.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:00.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:00.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:00.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:00.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:00.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:00.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:01.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:01.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:01.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:01.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:01.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:01.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:01.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:01.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:01.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:01.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:01.696+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:01.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:01.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:01.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:01.568+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:02.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:02.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:02.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:02.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:02.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:02.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:02.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:02.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:02.734+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:02.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:02.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:02.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:02.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:03.261 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:03.234+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13797 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:03.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:03.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:03.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:03.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:03.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:03.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:03.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:03.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:03.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:03.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:03.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:03.521+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:03.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:03.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:03.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:03.748+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:04.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:04.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:04.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:04.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:04.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:04.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:04.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:04.735+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:04.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:04.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:04.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:04.522+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:05.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:05.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:05.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:05.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:05.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:05.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:05.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:05.558+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:05.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:05.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:05.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:05.766+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:06.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:06.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:06.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:06.097+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:06.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:06.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:06.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:06.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:06.809+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:06.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:06.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:06.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:06.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:07.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:07.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:07.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:07.053+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:07.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:07.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:07.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:07.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:07.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:07.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:07.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:07.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:08.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:07.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:07.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:07.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:07.848+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:08.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:08.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:08.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:08.033+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:08.637 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:08.237+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13892 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:08.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:08.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:08.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:08.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:08.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:08.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:08.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:08.637+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:09.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:09.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:09.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:09.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:09.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:08.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:08.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:08.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:08.894+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:09.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:09.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:09.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:09.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:09.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:09.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:09.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:09.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:09.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:09.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:09.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:09.903+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:10.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:10.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:10.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:10.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:10.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:10.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:10.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:10.035+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:10.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:10.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:10.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:10.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:10.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:10.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:10.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:10.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:11.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:11.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:11.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:11.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:11.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:11.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:11.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:11.084+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:11.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:11.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:11.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:11.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:11.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:11.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:11.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:11.676+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:12.233 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:12.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:12.234 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:12.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:12.234 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:12.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:12.234 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:12.041+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:12.234 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:11.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:12.234 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:11.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:12.234 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:11.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:12.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:11.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:12.715+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:12.715+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:12.715+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:12.716+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:12.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:12.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:12.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:12.903+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:13.361 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:13.238+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13986 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:13.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:13.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:13.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:13.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:13.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:13.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:13.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:13.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:13.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:13.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:13.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:13.668+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:13.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:13.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:13.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:13.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:13.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:13.891+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:13.991+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:14.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:14.672+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:14.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:14.672+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:14.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:14.672+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:14.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:14.672+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:14.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:14.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:14.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:15.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:14.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:14.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:14.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:14.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:14.967+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:15.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:15.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:15.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:15.924+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:15.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:15.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:15.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:15.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:15.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:15.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:15.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:15.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:16.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:16.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:16.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:16.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:16.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:16.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:16.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:16.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:16.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:16.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:16.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:16.984+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:17.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:17.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:17.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:17.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:17.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:17.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:17.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:17.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:18.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:18.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:18.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:18.033+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:17.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:17.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:17.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:17.911+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:18.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:18.240+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14079 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:18.715+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:18.715+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:18.715+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:18.716+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:18.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:18.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:18.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:18.882+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:19.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:19.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:19.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:19.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:20.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:19.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:20.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:19.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:20.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:19.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:20.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:19.879+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:20.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:19.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:20.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:19.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:20.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:19.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:20.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:19.749+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:20.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:20.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:20.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:20.048+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:20.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:20.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:20.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:20.749+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:20.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:20.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:20.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:20.897+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:21.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:21.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:21.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:21.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:21.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:21.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:21.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:21.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:21.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:21.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:21.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:21.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:21.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:21.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:21.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:21.708+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:21.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:21.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:21.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:21.866+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:22.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:22.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:22.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:22.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:22.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:22.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:22.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:23.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:22.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:23.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:22.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:23.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:22.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:23.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:22.862+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:23.065 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:22.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:23.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:22.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:23.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:22.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:23.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:22.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:23.366 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:23.241+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14171 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:23.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:23.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:23.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:23.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:23.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:23.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:23.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:23.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:23.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:23.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:23.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:23.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:23.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:23.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:23.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:23.698+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:23.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:23.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:23.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:23.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:23.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:23.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:23.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:23.856+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:24.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:24.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:24.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:24.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:24.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:24.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:24.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:24.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:24.848+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:24.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:24.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:24.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:24.698+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:25.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:25.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:25.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:25.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:25.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:25.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:25.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:25.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:25.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:25.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:25.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:25.868+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:25.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:25.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:25.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:25.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:26.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:26.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:26.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:26.102+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:27.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:26.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:27.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:26.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:27.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:26.863+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:27.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:26.864+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:27.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:26.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:27.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:26.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:27.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:26.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:27.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:26.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:27.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:27.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:27.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:27.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:27.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:27.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:27.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:27.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:27.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:27.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:27.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:27.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:27.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:27.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:27.731+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:28.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:27.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:28.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:27.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:28.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:27.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:28.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:27.848+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:28.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:28.243+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14261 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:28.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:28.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:28.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:28.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:28.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:29.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:28.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:29.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:28.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:29.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:28.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:29.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:28.808+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:29.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:28.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:29.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:28.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:29.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:28.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:29.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:28.740+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:29.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:29.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:29.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:29.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:30.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:29.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:30.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:29.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:30.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:29.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:30.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:29.790+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:30.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:29.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:30.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:29.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:30.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:29.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:30.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:29.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:30.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:30.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:30.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:30.154+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:30.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:30.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:30.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:30.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:30.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:30.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:30.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:30.717+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:30.718+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:31.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:31.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:31.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:31.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:31.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:31.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:31.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:31.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:31.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:31.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:31.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:31.799+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:31.724+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:31.724+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:31.724+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:31.725+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:32.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:32.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:32.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:32.131+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:33.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:32.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:33.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:32.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:33.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:32.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:33.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:32.843+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:33.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:32.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:33.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:32.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:33.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:32.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:33.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:32.764+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:33.366 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:33.244+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14350 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:33.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:33.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:33.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:33.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:33.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:33.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:33.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:33.084+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:34.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:33.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:34.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:33.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:34.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:33.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:34.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:33.813+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:34.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:33.885+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:34.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:33.885+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:34.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:33.885+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:34.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:33.886+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:34.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:34.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:34.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:34.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:35.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:34.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:35.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:34.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:35.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:34.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:35.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:34.895+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:35.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:34.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:35.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:34.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:35.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:34.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:35.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:34.780+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:35.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:35.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:35.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:35.104+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:36.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:35.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:36.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:35.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:36.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:35.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:36.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:35.871+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:36.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:35.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:36.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:35.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:36.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:35.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:36.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:35.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:36.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:36.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:36.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:36.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:37.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:36.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:37.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:36.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:37.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:36.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:37.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:36.749+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:37.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:36.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:37.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:36.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:37.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:36.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:37.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:36.861+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:37.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:37.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:37.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:37.091+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:37.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:37.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:37.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:37.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:37.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:37.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:37.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:37.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:38.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:38.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:38.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:38.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:37.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:37.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:37.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:37.905+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:38.246+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14439 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:39.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:38.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:39.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:38.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:39.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:38.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:39.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:38.893+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:39.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:38.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:39.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:38.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:39.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:38.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:39.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:38.743+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:39.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:39.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:39.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:39.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:40.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:39.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:40.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:39.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:40.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:39.923+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:40.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:39.924+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:40.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:39.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:40.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:39.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:40.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:39.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:40.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:39.776+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:40.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:40.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:40.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:40.104+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:41.102 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:40.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:41.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:40.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:41.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:40.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:41.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:40.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:41.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:40.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:41.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:40.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:41.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:40.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:41.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:40.777+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:41.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:41.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:41.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:41.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:41.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:41.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:41.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:41.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:41.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:41.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:41.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:41.765+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:41.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:41.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:41.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:41.957+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:42.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:42.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:42.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:42.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:43.056 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:42.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:43.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:42.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:43.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:42.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:43.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:42.756+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:43.057 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:42.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:43.057 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:42.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:43.058 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:42.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:43.058 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:42.913+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:43.372 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:43.248+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14527 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:43.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:43.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:43.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:43.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:43.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:43.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:43.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:43.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:44.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:43.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:44.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:43.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:44.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:43.948+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:44.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:43.949+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:44.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:43.724+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:44.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:43.724+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:44.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:43.724+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:44.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:43.725+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:44.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:44.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:44.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:44.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:44.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:44.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:44.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:44.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:45.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:44.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:45.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:44.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:45.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:44.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:45.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:44.970+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:45.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:44.721+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:45.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:44.721+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:45.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:44.721+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:45.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:44.722+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:45.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:45.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:45.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:45.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:45.099+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:45.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:45.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:45.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:45.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:45.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:45.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:45.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:45.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:45.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:45.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:45.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:45.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:46.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:46.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:46.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:46.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:46.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:46.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:46.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:46.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:46.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:46.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:46.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:46.724+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:47.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:47.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:47.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:47.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:47.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:46.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:46.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:46.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:46.976+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:48.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:47.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:48.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:47.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:48.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:47.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:48.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:47.933+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:48.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:47.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:47.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:47.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:48.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:47.773+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:48.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:48.249+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14616 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:48.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:48.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:48.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:48.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:48.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:48.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:48.375 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:48.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:49.058 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:48.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:49.058 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:48.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:49.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:48.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:49.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:48.748+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:49.059 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:48.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:49.059 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:48.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:49.059 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:48.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:49.059 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:48.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:49.058+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:49.058+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:49.058+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:49.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:49.058+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:50.066 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:49.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:50.066 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:49.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:50.066 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:49.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:50.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:49.874+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:50.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:49.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:50.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:49.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:50.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:49.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:50.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:49.776+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:50.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:50.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:50.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:50.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:50.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:50.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:50.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:50.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:50.900+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:51.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:51.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:51.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:51.042+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:50.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:50.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:50.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:50.823+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:51.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:51.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:51.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:51.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:51.845+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:51.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:51.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:51.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:51.914+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:52.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:52.991+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:52.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:52.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:52.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:52.918+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:52.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:52.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:52.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:52.893+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:53.577 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:53.250+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14705 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:53.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:53.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:53.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:54.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:53.954+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:53.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:53.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:53.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:53.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:53.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:53.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:53.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:53.884+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:54.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:54.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:54.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:54.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:54.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:54.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:54.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:54.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:54.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:54.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:54.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:54.960+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:56.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:56.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:56.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:56.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:56.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:55.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:55.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:55.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:55.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:55.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:55.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:55.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:55.948+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:56.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:56.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:56.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:56.941+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:57.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:57.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:57.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:57.036+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.002+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.002+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.002+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.002+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:58.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:57.995+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:58.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:58.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:58.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:58.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:57.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:57.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:57.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:57.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:16:58.252+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14794 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:16:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:58.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:58.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:58.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:16:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:16:58.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:59.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:59.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:59.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:16:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:16:59.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:16:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:58.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:58.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:58.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:16:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:58.953+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:00.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:00.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:00.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:00.030+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:00.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:00.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:00.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:00.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:59.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:59.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:59.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:16:59.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:01.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:01.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:01.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:01.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:01.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:01.340 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:01.061+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:01.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:00.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:01.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:00.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:01.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:00.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:01.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:00.965+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:01.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:01.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:01.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:01.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:01.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:01.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:01.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:01.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:02.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:02.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:02.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:02.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:02.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:02.103+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:02.104+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:01.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:01.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:01.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:01.960+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:02.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:02.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:02.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:02.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:03.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:03.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:03.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:03.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:03.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:03.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:03.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:03.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:03.256 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:02.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:03.256 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:02.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:03.256 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:02.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:03.256 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:02.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:03.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:03.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:03.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:03.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:03.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:03.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:03.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:03.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:03.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:03.254+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14882 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-20T08:17:03.884 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-20T08:17:03.884 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T08:17:03.884 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:17:03.900 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-02-20T08:17:03.901 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:17:03.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-20T08:17:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:04.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:04.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:04.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:04.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:04.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:04.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:04.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:04.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:04.088+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:04.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:04.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:04.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:04.010+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:05.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:05.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:05.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:05.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:05.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:05.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:05.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:05.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:05.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:05.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:05.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:05.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:05.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:05.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:05.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:05.105+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:06.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:06.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:06.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:06.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:06.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:06.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:06.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:06.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.040+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:07.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:07.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:07.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:07.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:07.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:07.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:07.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:07.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:07.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:07.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:06.996+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:08.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:08.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:08.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:08.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:08.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:08.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:08.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:08.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:08.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:08.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:08.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:08.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:08.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:08.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.006+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:08.256+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14971 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:09.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:09.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:09.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:09.036+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:08.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:09.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:09.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:09.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:09.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:10.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:10.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:10.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:10.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:10.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:10.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:10.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:10.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:10.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:10.068+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:10.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:10.068+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:10.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:10.068+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:10.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:10.068+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:09.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:09.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:09.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:09.942+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:11.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:11.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:11.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:11.095+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:11.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:11.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:11.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:11.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:11.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:10.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:10.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:10.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:10.923+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:12.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:12.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:12.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:12.101+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:12.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:12.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:12.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:12.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:11.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:11.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:11.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:11.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:13.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:13.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:13.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:13.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:13.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:13.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:13.082+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:13.050+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:13.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:13.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:13.052+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:12.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:12.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:12.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:12.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:13.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:13.257+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15060 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:14.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:14.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:14.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:14.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:14.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:14.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:14.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:14.125+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:13.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:13.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:13.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:13.978+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:14.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:14.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:14.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:14.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:15.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:15.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:15.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:15.032+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:15.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:15.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:15.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:15.102+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:16.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:16.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:16.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:16.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:16.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:15.995+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:16.990+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.039+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:16.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:16.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:16.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:16.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:18.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:18.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:18.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:18.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:17.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:18.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:17.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:18.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:17.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:18.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:17.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:18.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:17.977+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:18.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:17.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:18.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:17.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:18.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:17.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:18.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:17.971+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:18.259+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15149 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:18.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:18.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:18.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:19.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:18.957+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:18.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:18.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:18.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:18.943+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:19.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:19.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:19.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:19.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:19.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:20.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:19.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:19.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:19.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:20.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:20.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:20.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:20.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:19.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:19.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:19.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:19.956+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:21.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:21.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:21.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:21.007+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:21.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:21.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:21.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:21.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:21.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:20.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:20.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:20.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:20.952+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:22.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:21.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:21.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:21.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:21.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:22.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:22.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:22.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:22.038+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:23.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:23.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:23.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:23.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.980+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.980+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.980+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:22.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:22.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:22.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:22.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:22.938+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:23.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:23.260+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15237 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:23.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:23.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:23.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:24.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:23.964+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:23.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:23.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:23.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:23.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:24.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:24.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:24.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:24.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:25.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:24.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:25.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:24.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:25.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:24.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:25.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:24.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:24.874+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:24.874+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:24.874+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:24.874+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:25.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:25.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:25.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:25.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:26.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:26.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:26.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:26.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:26.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:25.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:25.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:25.924+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:25.925+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:25.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:25.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:25.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:25.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:27.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:27.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:27.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:27.037+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:26.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:26.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:26.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:26.912+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:26.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:26.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:26.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:26.810+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:28.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:28.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:28.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:28.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:27.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:27.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:27.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:27.923+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:27.812+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:27.812+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:27.812+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:27.812+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:28.263+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15326 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:29.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:28.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:29.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:28.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:29.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:28.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:29.049 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:28.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:29.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:28.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:29.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:28.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:29.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:28.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:29.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:28.777+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:29.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:29.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:29.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:29.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:30.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:29.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:30.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:29.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:30.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:29.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:30.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:29.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:30.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:29.730+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:30.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:29.730+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:30.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:29.730+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:30.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:29.731+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:30.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:30.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:30.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:30.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:30.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:30.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:30.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:30.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:30.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:30.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:30.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:30.698+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:31.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:31.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:31.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:31.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:31.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:31.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:31.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:31.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:31.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:30.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:31.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:30.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:31.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:30.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:31.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:30.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:31.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:31.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:31.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:31.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:31.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:31.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:31.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:31.710+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.018+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:32.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:32.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:32.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:32.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:32.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:32.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:32.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:32.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:32.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:32.945+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:33.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:33.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:33.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:33.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:32.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:33.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:33.264+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15413 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:33.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:33.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:33.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:33.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:33.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:33.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:33.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:33.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:33.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:33.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:33.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:33.901+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:33.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:33.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:33.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:33.978+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:35.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:34.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:34.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:34.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:34.892+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:34.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:34.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:34.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:34.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:34.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:34.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:34.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:34.948+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:35.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:35.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:35.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:35.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:35.715+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:35.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:35.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:35.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:35.942+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:35.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:35.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:35.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:36.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:35.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:36.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:36.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:36.699+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:36.700+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:37.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:36.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:37.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:36.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:37.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:36.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:37.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:36.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:36.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:36.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:36.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:36.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:37.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:37.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:37.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:37.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:37.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:37.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:37.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:37.684+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:37.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:37.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:37.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:37.969+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:38.266+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15498 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:38.997 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:38.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:38.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:38.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:38.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:38.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:38.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:38.732+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:39.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:39.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:39.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:39.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:38.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:40.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:39.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:40.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:39.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:40.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:39.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:40.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:39.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:40.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:39.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:40.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:39.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:40.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:39.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:40.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:39.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:40.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:40.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:40.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:40.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:40.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:40.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:40.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:40.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:41.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:40.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:41.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:40.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:41.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:40.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:41.060 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:40.963+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:41.061 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:40.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:41.061 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:40.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:41.061 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:40.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:41.061 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:40.794+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:41.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:41.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:41.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:41.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:41.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:41.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:41.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:41.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:41.938+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:41.830+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:41.830+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:41.830+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:41.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:42.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:42.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:42.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:42.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:43.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:43.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:43.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:43.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:43.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:43.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:43.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:43.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:42.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:42.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:42.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:42.967+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:42.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:42.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:42.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:42.792+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:43.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:43.267+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15584 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:43.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:43.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:43.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:43.809+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:44.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:44.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:44.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:44.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:44.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:44.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:44.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:44.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:44.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:44.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:44.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:44.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:45.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:45.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:45.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:45.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:45.057+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:44.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:44.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:44.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:44.823+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.026+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:46.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:45.995+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:46.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:46.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:46.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:46.013+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:45.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:45.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:45.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:45.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:46.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:46.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:46.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:46.832+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:47.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:47.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:47.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:47.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:47.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:47.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:47.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:47.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:48.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:48.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:48.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:48.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:48.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:48.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:48.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:48.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:48.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:48.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:48.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:48.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:47.866+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:47.866+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:47.866+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:47.867+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:48.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:48.269+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15673 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:48.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:48.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:48.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:48.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:49.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:49.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:49.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:49.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:50.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:49.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:50.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:49.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:50.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:49.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:50.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:49.861+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:50.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:50.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:50.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:50.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:49.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:50.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:50.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:50.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:50.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:50.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:50.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:50.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:50.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:50.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:50.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:51.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:50.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:51.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:50.993+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:50.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:50.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:50.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:50.831+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.010+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:51.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:51.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:51.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:51.825+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:51.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:51.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:51.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:51.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:51.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:53.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:53.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:53.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:53.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:53.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:53.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:53.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:53.042+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:52.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:52.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:52.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:52.808+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:52.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:52.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:52.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:52.980+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:53.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:53.271+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15760 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:53.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:53.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:53.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:53.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:53.807+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:54.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:54.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:54.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:54.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:54.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:54.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:54.007+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:54.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:54.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:54.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:54.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:54.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:54.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:54.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:54.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:55.066 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:54.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:55.066 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:54.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:55.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:54.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:55.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:54.813+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:55.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:55.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:55.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:55.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:55.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:55.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:55.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:55.012+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:55.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:55.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:55.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:55.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:56.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:55.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:56.110 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:55.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:56.110 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:55.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:56.110 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:55.827+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:56.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:56.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:56.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:56.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:56.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:56.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:56.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:56.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:56.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:56.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:56.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:56.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:57.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:56.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:57.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:56.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:57.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:56.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:57.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:56.832+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:57.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:57.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:57.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:57.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:57.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:57.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:57.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:57.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:57.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:57.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:57.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:57.128+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:58.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:57.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:58.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:57.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:58.072 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:57.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:58.072 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:57.790+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:58.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:17:58.273+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15849 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:17:58.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:58.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:58.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:58.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:58.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:58.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:58.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:58.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:58.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:58.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:58.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:58.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:58.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:58.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:58.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:58.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:59.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:58.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:59.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:58.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:59.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:58.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:59.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:58.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:59.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:59.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:59.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:17:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:17:59.089+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:59.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:59.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:59.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:17:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:17:59.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:17:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:59.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:59.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:59.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:17:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:17:59.707+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:00.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:00.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:00.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:00.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:00.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:00.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:00.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:00.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:00.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:00.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:00.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:00.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:01.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:01.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:01.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:01.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:01.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:01.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:01.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:01.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:01.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:01.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:01.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:01.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:01.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:02.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:02.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:02.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:02.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:02.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:02.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:02.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:02.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:03.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:02.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:03.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:02.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:03.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:02.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:03.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:02.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:03.399 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:03.275+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15935 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:03.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:03.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:03.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:03.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:03.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:03.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:03.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:03.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:03.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:03.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:03.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:03.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:03.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:03.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:04.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:03.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:04.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:03.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:04.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:03.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:04.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:03.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:04.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:04.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:04.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:04.106+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:04.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:04.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:04.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:04.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:05.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:04.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:05.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:04.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:05.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:04.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:05.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:04.795+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:05.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:05.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:05.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:05.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:05.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:05.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:05.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:05.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:05.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:05.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:05.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:05.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:06.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:05.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:06.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:05.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:06.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:05.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:06.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:05.753+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:06.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:06.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:06.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:06.140+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:06.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:06.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:06.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:06.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:06.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:07.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:06.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:07.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:06.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:07.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:06.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:07.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:06.777+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:07.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:07.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:07.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:07.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:07.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:07.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:07.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:07.018+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:07.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:07.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:07.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:07.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:07.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:07.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:07.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:07.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:08.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:08.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:08.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:08.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:08.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:08.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:08.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:08.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:08.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:08.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:08.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:08.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:08.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:08.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:08.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:08.007+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:08.276+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16024 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:09.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:08.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:09.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:08.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:09.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:08.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:09.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:08.730+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:09.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:09.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:09.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:09.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:09.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:09.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:09.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:09.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:09.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:09.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:09.040+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:10.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:09.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:10.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:09.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:10.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:09.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:10.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:09.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:10.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:10.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:10.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:10.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:10.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:10.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:10.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:10.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:10.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:10.709+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:11.261 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:11.262 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:11.262 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:11.262 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:11.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:11.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:11.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:11.263 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:10.982+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:11.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:11.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:11.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:11.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:11.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:11.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:11.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:11.733+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:11.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:11.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:11.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:11.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:11.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:12.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:12.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:12.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:12.781+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:12.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:12.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:12.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:12.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:12.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:12.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:12.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:12.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:13.583 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:13.278+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16111 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:13.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:13.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:13.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:13.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:13.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:13.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:13.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:13.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:13.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:13.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:13.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:13.968+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:13.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:13.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:13.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:13.985+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:15.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:14.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:14.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:14.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:14.951+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:14.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:14.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:14.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:14.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:15.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:15.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:15.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:15.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:15.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:15.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:15.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:15.819+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:15.986+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:16.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:16.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:16.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:17.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:16.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:16.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:16.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:16.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:16.802+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:16.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:16.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:16.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:16.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:18.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:17.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:18.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:17.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:17.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:17.963+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:18.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:17.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:18.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:17.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:18.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:17.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:18.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:17.809+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:17.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:17.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:17.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:17.920+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:18.280+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16200 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:18.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:18.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:18.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:18.855+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:18.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:18.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:18.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:18.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:19.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:18.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:18.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:18.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:18.937+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:20.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:20.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:20.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:20.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:20.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:20.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:20.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:20.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:20.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:19.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:20.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:19.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:20.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:19.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:20.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:19.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:20.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:19.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:20.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:19.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:20.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:19.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:20.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:19.934+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:20.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:20.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:20.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:20.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:20.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:20.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:20.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:20.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:21.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:21.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:21.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:21.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:21.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:21.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:21.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:21.917+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:21.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:21.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:21.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:21.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:22.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:22.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:22.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:22.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:23.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:23.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:23.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:23.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:23.084+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:22.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:22.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:22.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:22.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:22.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:22.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:23.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:22.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:23.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:22.934+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:23.549 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:23.281+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16286 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:24.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:23.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:24.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:23.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:23.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:23.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:24.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:24.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.044+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:23.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:23.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:23.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:23.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:24.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:24.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:24.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:25.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:24.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:24.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:24.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:24.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:24.945+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:24.997+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:26.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:25.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:26.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:25.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:26.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:25.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:26.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:25.973+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:25.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:25.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:25.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:25.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:25.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:25.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:25.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:25.984+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:26.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:27.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:26.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:27.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:26.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:27.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:26.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:27.226 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:27.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:27.226 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:27.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:27.226 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:27.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:27.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:27.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:27.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:26.965+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:27.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:26.965+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:27.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:26.965+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:27.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:26.966+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:28.284 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:28.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:28.284 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:28.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:28.284 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:28.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:28.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:28.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:28.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:28.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:28.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:28.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:28.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:28.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:28.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:28.030+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:28.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:27.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:28.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:27.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:28.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:27.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:28.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:27.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:28.283+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16375 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:29.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:29.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:29.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:29.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:29.066+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:29.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:29.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:29.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:29.036+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:28.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:28.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:28.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:28.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:30.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:30.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:30.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:30.008+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:29.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:29.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:29.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:29.971+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:30.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:30.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:30.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:30.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:31.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:31.041+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:31.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:31.041+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:31.041+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:31.042+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:31.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:31.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:31.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:31.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:30.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:30.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:30.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:30.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:32.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:32.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:32.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:32.081+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:32.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:32.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:32.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:32.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:32.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:32.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:32.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:32.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:31.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:31.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:31.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:31.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:33.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:32.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:33.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:32.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:33.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:32.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:33.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:32.999+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:33.408 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:33.285+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16462 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:33.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:33.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:33.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:33.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:33.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:33.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:33.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:33.085+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:33.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:33.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:33.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:33.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:33.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:33.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:33.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:33.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:34.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:34.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:34.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:34.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:34.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:34.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:34.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:34.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:34.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:34.126+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:34.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:34.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:34.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:34.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:35.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:35.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:35.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:35.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:35.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:35.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:35.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:35.226+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:35.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:35.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:35.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:35.038+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:36.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:36.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:36.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:36.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:36.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:36.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:36.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:36.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:36.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:36.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:36.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:36.169+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:36.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:36.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:36.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:36.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:37.283 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:37.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:37.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:37.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:37.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:37.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:37.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:37.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:37.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:37.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:37.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:37.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:37.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:37.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:37.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:37.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:37.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:37.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:37.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:37.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:38.411 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:38.286+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16550 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:38.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:38.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:38.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:38.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:38.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:38.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:38.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:38.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:38.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:38.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:38.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:38.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:38.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:38.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:38.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:38.168+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:38.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:38.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:38.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:38.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:38.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:38.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:38.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:38.076+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:39.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:39.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:39.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:39.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:39.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:39.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:39.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:39.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:39.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:39.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:39.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:39.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:39.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:39.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:39.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:39.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:39.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:39.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:39.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:39.198+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:40.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:40.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:40.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:40.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:40.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:40.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:40.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:40.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:40.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:40.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:40.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:40.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:41.228+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:41.228+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:41.228+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:41.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:41.229+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:41.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:41.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:41.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:41.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:41.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:41.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:41.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:41.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:42.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:42.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:42.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:42.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:42.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:42.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:42.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:42.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:42.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:42.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:42.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:42.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:42.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:42.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:42.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:42.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:42.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:42.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:42.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:42.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:43.414 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:43.288+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16636 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:43.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:43.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:43.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:43.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:43.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:43.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:43.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:43.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:43.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:43.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:43.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:43.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:43.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:43.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:43.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:43.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:43.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:43.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:43.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:43.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:43.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:43.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:43.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:43.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:44.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:44.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:44.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:44.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:44.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:44.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:44.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:44.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:44.163+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:44.163+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:44.163+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:44.164+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:45.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:45.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:45.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:45.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:45.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:45.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:45.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:45.212+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:45.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:45.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:45.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:45.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:46.478 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:46.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:46.479 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:46.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:46.479 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:46.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:46.479 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:46.200+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:46.479 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:46.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:46.479 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:46.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:46.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:46.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:46.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:46.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:46.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:46.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:46.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:46.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:46.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:46.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:46.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:46.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:47.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:47.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:47.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:47.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:47.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:47.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:47.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:47.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:47.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:47.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:47.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:47.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:47.223+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:48.415 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:48.290+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16725 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:48.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:48.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:48.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:48.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:48.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:48.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:48.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:48.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:48.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:48.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:48.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:48.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:48.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:48.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:48.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:48.137+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:48.417 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:48.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:48.418 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:48.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:48.418 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:48.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:48.418 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:48.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:49.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:49.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:49.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:49.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:49.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:49.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:49.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:49.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:49.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:49.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:49.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:49.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:49.201+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:49.201+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:49.201+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:49.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:49.202+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:50.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:50.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:50.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:50.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:50.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:50.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:50.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:50.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:50.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:50.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:50.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:50.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:50.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:51.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:51.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:51.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:51.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:51.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:51.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:51.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:51.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:51.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:51.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:51.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:51.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:52.305 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:52.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:52.305 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:52.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:52.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:52.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:52.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:52.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:52.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:52.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:52.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:52.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:52.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:52.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:52.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:52.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:52.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:52.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:52.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:52.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:52.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:53.349 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:53.292+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16811 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:53.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:53.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:53.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:53.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:53.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:53.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:53.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:53.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:53.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:53.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:53.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:53.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:53.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:53.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:53.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:53.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:53.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:53.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:53.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:53.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:53.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:53.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:53.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:53.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:54.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:54.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:54.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:54.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:54.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:54.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:54.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:54.178+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:54.387 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:54.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:54.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:54.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:54.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:54.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:54.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:54.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:54.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:54.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:54.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:54.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:55.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:55.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:55.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:55.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:55.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:55.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:55.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:55.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:55.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:55.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:55.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:55.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:55.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:56.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:56.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:56.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:56.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:56.337+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:56.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:56.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:56.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:56.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:56.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:56.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:56.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:56.162+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:57.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:57.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:57.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:57.163+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:57.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:57.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:57.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:57.141+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:57.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:57.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:57.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:57.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:58.419 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:18:58.293+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16899 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:18:58.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:58.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:58.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:58.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:58.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:58.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:58.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:58.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:58.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:58.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:58.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:58.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:58.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:58.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:58.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:58.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:58.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:58.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:58.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:58.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:58.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:58.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:58.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:58.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:59.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:59.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:59.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:18:59.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:18:59.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:59.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:59.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:59.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:18:59.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:59.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:59.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:59.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:18:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:18:59.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:00.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:00.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:00.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:00.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:00.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:00.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:00.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:00.168+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:00.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:00.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:00.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:00.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:00.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:00.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:00.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:00.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:00.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:00.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:01.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:01.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:01.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:01.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:01.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:01.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:01.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:01.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:01.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:01.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:01.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:01.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:02.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:02.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:02.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:02.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:02.126+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:02.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:02.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:02.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:02.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:02.220+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:03.419 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:03.295+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16984 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:03.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:03.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:03.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:03.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:03.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:03.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:03.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:03.224+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:03.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:03.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:03.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:03.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:03.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:03.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:03.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:03.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:03.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:03.125+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-20T08:19:04.031 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-20T08:19:04.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T08:19:04.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-20T08:19:04.072 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.072 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/tpci.mod.c: Cannot open: Input/output error 2024-02-20T08:19:04.072 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/tpci.h: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/tpci.c: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/st_tpci.h: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/pci-exp-tests-HOWTO: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci: Cannot utime: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci: Cannot mkdir: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/tpci_ki.c: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/user_tpci.c: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.073 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/user_tpci.h: Cannot open: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci: Cannot utime: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb: Cannot mkdir: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb: Cannot mkdir: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/st_tusb.h: Cannot open: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/tusb.h: Cannot open: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/README: Cannot open: Input/output error 2024-02-20T08:19:04.074 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/tusb.c: Cannot open: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb: Cannot mkdir: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/user_tusb.c: Cannot open: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/tusb_ki.c: Cannot open: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/user_tusb.h: Cannot open: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base: Cannot mkdir: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base: Cannot mkdir: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.075 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/user_tbase.c: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/user_tbase.h: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/tbase_ki.c: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase: Cannot mkdir: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/tbase.mod.c: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/tbase.c: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/tbase.h: Cannot open: Input/output error 2024-02-20T08:19:04.076 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/load_tbase.sh: Cannot open: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/str_mod.h: Cannot open: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework: Cannot mkdir: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space: Cannot mkdir: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/user_tmod.c: Cannot open: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/tmod_ki.c: Cannot open: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/user_tmod.h: Cannot open: Input/output error 2024-02-20T08:19:04.077 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space: Cannot mkdir: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/load_tmod.sh: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/Makefile: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/tmod.h: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/str_mod.h: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/tmod.c: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/tmod.mod.c: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/README: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/README: Cannot open: Input/output error 2024-02-20T08:19:04.078 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l: Cannot mkdir: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space: Cannot mkdir: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_QUERYCTRL.c: Cannot open: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_EXT_CTRLS.h: Cannot open: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUM_FRAMESIZES.c: Cannot open: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_PARM.h: Cannot open: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/v4l2_test.c: Cannot open: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_QUERYMENU.h: Cannot open: Input/output error 2024-02-20T08:19:04.079 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUMINPUT.c: Cannot open: Input/output error 2024-02-20T08:19:04.080 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_QUERYBUF.c: Cannot open: Input/output error 2024-02-20T08:19:04.080 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_INPUT.h: Cannot open: Input/output error 2024-02-20T08:19:04.080 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/dev_video.h: Cannot open: Input/output error 2024-02-20T08:19:04.080 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_FREQUENCY.h: Cannot open: Input/output error 2024-02-20T08:19:04.080 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_OUTPUT.h: Cannot open: Input/output error 2024-02-20T08:19:04.080 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_FREQUENCY.c: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_AUDOUT.h: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUMAUDOUT.h: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUMAUDIO.h: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_CTRL.h: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc: Cannot mkdir: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc/results.html: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc/index.html: Cannot open: Input/output error 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:tar: 2024-02-20T08:19:04.081 INFO:tasks.workunit.client.0.smithi055.stderr:gzip: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc/spec: Cannot mkdir: Input/output error 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:stdin: Stale file handle 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:tar: Unexpected EOF in archive 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers: Cannot utime: Input/output error 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-02-20T08:19:04.082 INFO:tasks.workunit.client.0.smithi055.stderr:tar: Error is not recoverable: exiting now 2024-02-20T08:19:04.083 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-02-20T08:19:04.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T08:19:04.084 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-20T08:19:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-20T08:19:04.087 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-20T08:19:04.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-20T08:19:04.151 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-02-20T08:19:04.152 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:19:04.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:04.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:04.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:04.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:04.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:04.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:04.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:04.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:04.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:04.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:04.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:04.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:04.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:04.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:04.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:04.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:04.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:04.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:04.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:04.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:04.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:04.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:04.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:04.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:05.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:05.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:05.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:05.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:05.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:05.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:05.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:05.253+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:05.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:05.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:05.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:05.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:06.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:06.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:06.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:06.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:06.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:06.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:06.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:06.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:06.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:06.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:06.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:06.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:06.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:07.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:07.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:07.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:07.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:07.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:07.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:07.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:07.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:07.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:07.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:07.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:07.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:07.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:08.319 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:08.297+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17072 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:08.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:08.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:08.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:08.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:08.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:08.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:08.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:08.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:08.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:08.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:08.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:08.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:08.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:08.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:08.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:08.280+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:08.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:08.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:08.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:08.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:08.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:08.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:08.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:08.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:09.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:09.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:09.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:09.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:09.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:09.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:09.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:09.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:09.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:09.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:09.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:09.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:09.147+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:10.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:10.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:10.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:10.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:10.317 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:10.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:10.317 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:10.121+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:10.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:10.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:10.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:10.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:10.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:10.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:10.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:10.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:10.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:10.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:10.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:10.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:11.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:11.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:11.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:11.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:11.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:11.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:11.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:11.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:11.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:11.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:11.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:11.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:11.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:11.080+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:12.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:12.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:12.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:12.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:12.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:12.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:12.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:12.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:12.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:12.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:12.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:12.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:12.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:12.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:12.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:12.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:12.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:13.421 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:13.299+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17150 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:13.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:13.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:13.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:13.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:13.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:13.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:13.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:13.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:13.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:13.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:13.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:13.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:13.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:13.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:13.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:13.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:13.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:13.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:13.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:13.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:13.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:13.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:13.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:13.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:14.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:14.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:14.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:14.194+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:14.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:14.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:14.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:14.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:14.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:14.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:14.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:14.270+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:15.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:15.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:15.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:15.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:15.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:15.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:15.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:15.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:15.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:15.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:15.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:15.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:16.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:16.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:16.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:16.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:16.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:16.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:16.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:16.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:16.177+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:16.177+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:16.177+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:16.177+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:16.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:16.163+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:16.163+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:16.163+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:16.164+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:17.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:17.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:17.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:17.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:17.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:17.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:17.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:17.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:17.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:17.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:17.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:17.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:18.422 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:18.300+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17238 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:18.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:18.162+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:18.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:18.162+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:18.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:18.162+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:18.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:18.163+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:18.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:18.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:18.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:18.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:18.305+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:18.424 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:18.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:18.424 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:18.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:18.425 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:18.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:18.425 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:18.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:19.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:19.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:19.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:19.169+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:19.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:19.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:19.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:19.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:19.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:19.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:19.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:19.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:20.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:20.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:20.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:20.140+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:20.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:20.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:20.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:20.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:20.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:20.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:20.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:20.148+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:21.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:21.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:21.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:21.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:21.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:21.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:21.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:21.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:21.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:21.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:21.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:21.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:21.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:21.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:21.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:21.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:21.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:22.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:22.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:22.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:22.336+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:22.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:22.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:22.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:22.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:22.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:22.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:22.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:22.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:23.376 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:23.302+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17324 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:23.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:23.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:23.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:23.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:23.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:23.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:23.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:23.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:23.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:23.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:23.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:23.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:23.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:23.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:23.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:23.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:23.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:23.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:23.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:23.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:23.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:24.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:24.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:24.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:24.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:24.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:24.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:24.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:24.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:24.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:24.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:24.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:24.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:24.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:24.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:24.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:24.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:24.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:24.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:24.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:24.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:25.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:25.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:25.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:25.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:25.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:25.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:25.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:25.182+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:25.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:25.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:25.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:25.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:25.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:25.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:25.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:25.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:25.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:25.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:25.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:25.358+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:26.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:26.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:26.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:26.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:26.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:26.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:26.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:26.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:26.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:26.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:26.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:26.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:26.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:26.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:26.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:26.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:27.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:27.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:27.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:27.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:27.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:27.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:27.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:27.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:27.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:27.197+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:27.197+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:27.197+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:27.198+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:28.427 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:28.304+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17413 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:28.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:28.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:28.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:28.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:28.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:28.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:28.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:28.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:28.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:28.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:28.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:28.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:28.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:28.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:28.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:28.224+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:28.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:28.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:28.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:28.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:28.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:28.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:28.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:28.088+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:29.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:29.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:29.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:29.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:29.089+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:29.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:29.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:29.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:29.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:29.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:29.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:29.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:29.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:29.226+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:30.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:30.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:30.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:30.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:30.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:30.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:30.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:30.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:30.274+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:30.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:30.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:30.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:30.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:31.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:31.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:31.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:31.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:31.085+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:31.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:31.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:31.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:31.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:31.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:31.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:31.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:31.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:32.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:32.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:32.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:32.210+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:32.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:32.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:32.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:32.125+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:32.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:32.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:32.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:32.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:32.304+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:33.428 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:33.306+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17500 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:33.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:33.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:33.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:33.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:33.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:33.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:33.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:33.151+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:33.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:33.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:33.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:33.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:33.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:33.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:33.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:33.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:33.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:33.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:33.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:33.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:33.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:33.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:33.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:33.253+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:34.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:34.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:34.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:34.141+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:34.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:34.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:34.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:34.322+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:34.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:34.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:34.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:34.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:35.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:35.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:35.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:35.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:35.125+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:35.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:35.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:35.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:35.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:35.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:35.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:35.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:35.281+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:36.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:36.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:36.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:36.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:36.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:36.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:36.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:36.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:36.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:36.237+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:36.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:36.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:36.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:36.159+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:37.229 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:37.137+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:37.230 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:37.137+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:37.230 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:37.137+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:37.230 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:37.138+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:37.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:37.281+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:37.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:37.281+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:37.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:37.281+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:37.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:37.282+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:37.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:37.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:37.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:37.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:37.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:37.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:37.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:37.306+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:38.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:38.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:38.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:38.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:38.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:38.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:38.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:38.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:38.308+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17589 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:38.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:38.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:38.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:38.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:38.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:38.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:38.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:38.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:38.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:38.311+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:39.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:39.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:39.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:39.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:39.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:39.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:39.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:39.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:39.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:39.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:39.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:39.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:39.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:40.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:40.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:40.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:40.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:40.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:40.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:40.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:40.274+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:40.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:40.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:40.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:40.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:41.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:41.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:41.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:41.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:41.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:41.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:41.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:41.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:41.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:41.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:41.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:41.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:42.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:42.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:42.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:42.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:42.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:42.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:42.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:42.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:42.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:42.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:42.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:42.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:42.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:43.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:43.309+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17675 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:43.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:43.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:43.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:43.303+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:43.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:43.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:43.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:43.323+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:43.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:43.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:43.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:43.187+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:44.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:44.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:44.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:44.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:44.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:44.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:44.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:44.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:44.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:44.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:44.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:44.282+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:45.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:45.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:45.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:45.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:45.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:45.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:45.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:45.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:45.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:45.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:45.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:45.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:45.137+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:46.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:46.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:46.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:46.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:46.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:46.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:46.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:46.277+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:46.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:46.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:46.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:46.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:46.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:46.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:46.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:46.115+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:47.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:47.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:47.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:47.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:47.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:47.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:47.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:47.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:47.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:47.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:47.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:47.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:47.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:47.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:47.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:47.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:47.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:47.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:47.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:47.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:48.312 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:48.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:48.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:48.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:48.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:48.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:48.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:48.225+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:48.313 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:48.313 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:48.313 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:48.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:48.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:48.311+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17764 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:48.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:48.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:48.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:48.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:48.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:48.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:48.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:49.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:48.992+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:49.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:49.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:49.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:49.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:49.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:49.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:49.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:49.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:50.242 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:49.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:50.242 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:49.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:50.243 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:49.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:50.243 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:49.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:50.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:50.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:50.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:50.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:50.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:50.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:50.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:50.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:50.242+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:51.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:51.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:51.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:51.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:51.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:51.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:51.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:51.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:51.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:50.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:51.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:50.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:51.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:50.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:51.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:50.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:51.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:51.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:51.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:51.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:51.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:51.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:51.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:51.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:51.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:51.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:51.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:52.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:51.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:52.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:52.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:52.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:52.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:52.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:52.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:52.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:52.437+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:52.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:52.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:52.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:52.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:52.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:52.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:52.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:52.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:53.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:53.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:53.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:53.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:53.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:53.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:53.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:53.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:53.590 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:53.313+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17851 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:53.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:53.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:53.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:53.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:53.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:53.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:53.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:53.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:53.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:53.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:53.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:53.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:53.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:53.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:53.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:53.287+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:54.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:54.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:54.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:54.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:54.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:54.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:54.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:54.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:54.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:54.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:54.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:54.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:54.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:54.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:54.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:54.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:54.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:55.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:55.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:55.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:55.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:55.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:55.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:55.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:55.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:55.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:55.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:55.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:55.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:55.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:55.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:55.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:55.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:55.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:55.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:55.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:55.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:55.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:55.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:55.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:55.322+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:56.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:56.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:56.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:56.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:56.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:56.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:56.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:56.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:56.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:56.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:56.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:56.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:56.442+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:56.442+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:56.442+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:56.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:57.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:57.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:57.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:57.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:57.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:57.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:57.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:57.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:57.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:57.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:57.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:57.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:57.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:57.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:57.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:57.437+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:58.316 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:58.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:58.316 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:58.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:58.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:58.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:58.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:58.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:19:58.315+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17940 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:19:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:58.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:58.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:58.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:58.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:58.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:58.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:58.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:58.485+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:59.387 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:59.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:59.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:59.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:59.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:59.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:19:59.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:19:59.010+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:59.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:59.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:59.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:19:59.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:19:59.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:19:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:59.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:59.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:59.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:19:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:19:59.388+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:00.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:00.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:00.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:00.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:00.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:00.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:00.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:00.001+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:00.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:00.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:00.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:00.427+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:00.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:00.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:00.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:00.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:01.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:01.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:01.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:01.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:01.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:01.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:01.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:01.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:01.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:01.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:01.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:01.463+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:01.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:01.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:01.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:01.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:01.399+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:02.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:02.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:02.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:02.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:02.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:02.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:02.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:02.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:02.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:02.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:02.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:02.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:02.444+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:03.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:03.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:03.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:03.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:03.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:03.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:03.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:03.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:03.592 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:03.316+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18026 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:03.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:03.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:03.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:03.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:03.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:03.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:03.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:03.495+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:03.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:03.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:03.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:03.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:03.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:03.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:03.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:03.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:04.428 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:04.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:04.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:04.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:04.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:04.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:04.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:04.038+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:04.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:04.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:04.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:04.429+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:04.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:04.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:04.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:04.462+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:05.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:05.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:05.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:05.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:05.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:05.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:05.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:05.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:05.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:05.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:05.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:05.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:05.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:05.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:06.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:06.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:06.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:06.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:06.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:06.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:06.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:06.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:06.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:06.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:06.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:06.460+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:06.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:06.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:06.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:06.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:07.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:07.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:07.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:07.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:07.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:07.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:07.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:07.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:07.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:07.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:07.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:07.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:07.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:07.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:08.308 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:08.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:08.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:08.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:08.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:08.036+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:08.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:08.037+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:08.318+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18115 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:08.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:08.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:08.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:08.556+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:08.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:08.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:08.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:08.308+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:09.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:09.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:09.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:09.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:09.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:09.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:09.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:09.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:09.052+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:09.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:09.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:09.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:09.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:09.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:10.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:10.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:10.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:10.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:10.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:10.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:10.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:10.243+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:10.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:10.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:10.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:10.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:10.512+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:11.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:11.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:11.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:11.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:11.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:11.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:11.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:11.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:11.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:11.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:11.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:11.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:11.198+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:11.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:11.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:11.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:11.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:11.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:11.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:11.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:11.465+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:12.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:12.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:12.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:12.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:12.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:12.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:12.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:12.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:12.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:12.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:12.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:12.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:12.504+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:13.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:13.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:13.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:13.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:13.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:13.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:13.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:13.223+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:13.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:13.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:13.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:13.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:13.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:13.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:13.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:13.069+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:13.592 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:13.320+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18202 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:13.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:13.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:13.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:13.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:13.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:13.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:13.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:13.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:14.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:14.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:14.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:14.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:14.225+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:14.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:14.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:14.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:14.055+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:14.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:14.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:14.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:14.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:15.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:15.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:15.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:15.208+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:15.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:15.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:15.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:15.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:15.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:15.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:15.531+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:15.532+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:16.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:16.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:16.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:16.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:16.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:16.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:16.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:16.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:16.188+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:16.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:16.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:16.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:16.494+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:17.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:17.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:17.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:17.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:17.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:17.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:17.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:17.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:17.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:17.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:17.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:17.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:18.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:18.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:18.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:18.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:18.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:18.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:18.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:18.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:18.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:18.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:18.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:18.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:18.321+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18291 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:18.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:18.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:18.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:18.460+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:19.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:19.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:19.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:19.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:19.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:19.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:19.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:19.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:19.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:19.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:19.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:19.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:19.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:19.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:19.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:19.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:19.508+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:20.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:20.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:20.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:20.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:20.168+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:20.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:20.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:20.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:20.050+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:20.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:20.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:20.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:20.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:20.488+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:21.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:21.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:21.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:21.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:21.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:21.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:21.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:21.208+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:21.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:21.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:21.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:21.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:21.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:21.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:21.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:21.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:21.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:21.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:21.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:21.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:22.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:22.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:22.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:22.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:22.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:22.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:22.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:22.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:22.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:22.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:22.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:22.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:22.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:22.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:50.434673+0000 (oldest deadline 2024-02-20T08:10:10.434673+0000) 2024-02-20T08:20:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:22.496+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:23.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:23.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:23.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:23.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:23.326 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:23.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:51.475309+0000 (oldest deadline 2024-02-20T08:10:11.475309+0000) 2024-02-20T08:20:23.326 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:23.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:23.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:23.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:23.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:23.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:23.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:23.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:23.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:23.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:23.593 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:23.324+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18377 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:23.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:23.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:24.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:24.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:24.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:24.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:24.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:24.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:24.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:24.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:24.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:24.062+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:24.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:25.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:25.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:25.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:25.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:25.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:25.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:25.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:25.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:25.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:25.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:25.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:25.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:26.372 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:26.174+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:26.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:26.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:26.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:26.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:26.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:26.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:26.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:26.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:26.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:26.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:27.410 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:27.204+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:27.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:27.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:27.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:27.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:27.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:27.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:27.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:27.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:27.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:27.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:28.415 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:28.326+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18465 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:28.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:28.194+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:28.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:28.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:28.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:28.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:28.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:28.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:28.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:28.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:28.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:28.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:29.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:29.224+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:29.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:29.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:29.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:29.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:29.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:29.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:29.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:29.115+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:29.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:29.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:30.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:30.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:30.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:30.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:30.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:30.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:30.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:30.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:09:54.235957+0000 (oldest deadline 2024-02-20T08:10:14.235957+0000) 2024-02-20T08:20:30.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:30.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:30.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:30.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:31.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:31.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:31.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:31.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:31.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:31.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:32.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:32.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:32.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:32.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:32.130+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:33.451 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:33.328+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18550 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:33.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:33.358+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:33.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:33.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:33.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:33.164+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:34.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:34.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:34.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:34.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:34.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:35.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:35.357+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:35.174+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:35.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:36.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:36.130+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:36.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:36.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:36.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:36.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:37.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:37.121+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:37.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:37.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:37.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:37.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:38.453 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:38.330+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18637 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:38.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:38.345+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:38.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:38.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:38.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:38.175+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:39.350+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:39.118+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:39.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:39.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:40.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:40.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:40.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:40.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:40.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:40.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:41.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:41.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:41.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:41.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:41.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:41.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:42.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:42.056+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:42.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:42.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:42.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:42.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:43.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:43.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:43.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:43.142+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:43.593 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:43.331+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18723 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:43.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:43.358+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:44.339+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:44.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:44.104+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:44.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:44.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:45.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:45.056+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:45.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:45.162+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:45.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:45.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:46.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:46.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:46.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:46.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:46.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:46.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:46.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:46.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:46.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:46.024+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:46.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:46.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:47.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:47.340 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:47.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:47.340 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:47.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:47.340 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:47.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:47.340 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:47.058+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:47.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:47.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:47.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:47.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:47.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:48.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:48.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:48.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:48.016+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:48.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:48.333+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18810 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:48.252+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:48.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:48.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:48.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:48.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:48.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:48.326+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:49.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:49.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:49.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:49.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:49.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:49.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:49.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:49.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:49.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:49.041+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:49.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:49.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:49.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:49.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:49.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:50.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:50.215+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:50.284 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:50.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:50.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:50.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:50.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:50.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:50.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:50.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:50.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:50.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:50.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:50.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:50.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:50.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:50.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:50.034+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:51.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:51.188+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:51.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:51.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:51.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:51.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.001+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:51.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:51.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:51.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:51.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:51.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:51.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:51.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:51.291+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:52.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:52.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:52.227 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:52.227 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:51.976+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:52.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:52.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:52.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:52.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:52.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:52.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:53.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:53.235+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:53.336 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:53.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:53.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:53.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.001+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:53.594 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:53.335+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18897 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:53.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:53.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:53.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:53.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:53.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:53.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:53.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:53.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:54.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:54.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:54.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:54.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:54.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:54.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:53.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:54.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:54.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:54.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:54.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:54.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:54.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:54.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:54.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:54.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:55.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:55.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:55.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:55.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:55.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:55.367+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:56.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:55.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:56.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:55.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:56.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:55.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:56.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:55.915+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:56.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:56.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:56.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:56.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:56.152+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:56.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:56.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:56.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:56.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:56.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:56.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:56.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:56.960+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:57.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:57.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:57.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:57.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:57.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:57.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:57.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:57.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:57.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:57.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:57.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:57.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:57.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:58.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:57.967+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:58.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:57.967+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:58.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:57.967+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:58.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:57.968+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:58.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:58.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:58.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:58.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:58.197+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:20:58.337+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18985 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:20:58.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:58.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:58.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:58.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:58.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:58.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:58.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:58.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:58.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:58.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:58.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:20:59.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:58.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:59.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:59.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:59.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:20:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:20:59.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:20:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:59.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:59.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:59.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:20:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:20:59.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:00.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:59.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:00.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:59.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:00.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:59.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:00.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:20:59.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:00.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:00.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:00.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:00.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:00.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:00.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:00.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:00.145+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:00.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:00.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:00.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:00.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:00.428+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:00.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:00.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:00.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:00.954+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:01.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:01.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:01.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:01.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:01.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:01.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:01.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:01.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:01.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:01.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:01.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:01.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:01.418+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:02.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:02.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:02.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:02.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:02.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:02.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:02.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:02.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:02.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:02.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:02.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:02.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:02.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:02.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:02.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:02.151+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:02.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:03.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:03.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:03.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:03.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:03.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:03.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:03.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:03.001+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:03.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:03.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:03.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:03.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:03.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:03.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:03.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:03.339+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19071 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:03.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:03.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:03.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:03.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:03.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:03.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:03.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:03.424+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:04.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:04.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:04.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:04.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:04.025+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:04.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:04.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:04.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:04.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:04.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:04.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:04.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:04.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:04.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:05.457 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:05.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:05.457 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:05.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:05.457 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:05.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:05.457 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:05.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:05.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:05.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:05.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:05.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:05.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:05.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:05.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:05.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:05.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:05.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:05.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:05.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:05.457+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:06.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:06.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:06.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:06.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:06.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:06.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:06.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:06.029+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:06.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:06.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:06.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:06.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:06.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:06.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:06.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:06.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:06.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:06.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:06.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:06.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:06.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:07.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:07.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:07.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:07.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:07.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:07.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:07.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:07.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:07.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:07.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:07.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:07.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:07.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:07.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:07.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:07.062+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:07.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:07.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:07.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:07.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:07.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:08.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:08.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:08.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:08.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:08.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:08.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:08.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:08.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:08.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:08.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:08.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:08.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:08.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:08.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:08.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:08.062+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:08.340+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19158 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:08.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:08.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:08.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:08.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:09.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:09.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:09.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:09.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:09.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:09.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:09.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:09.090+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:09.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:09.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:09.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:09.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:09.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:09.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:09.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:09.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:09.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:10.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:10.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:10.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:10.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:10.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:10.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:10.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:10.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:10.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:10.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:10.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:10.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:09.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:10.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:10.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:10.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:10.452+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:11.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:11.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:11.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:11.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:11.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:11.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:11.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:11.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:11.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:11.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:11.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:11.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:11.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:11.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:11.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:11.101+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:11.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:11.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:11.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:11.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:11.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:11.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:11.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:11.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.009+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:12.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:12.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:12.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:12.094+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:12.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:12.483+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:12.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:12.483+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:12.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:12.483+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:12.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:12.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:13.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:13.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:13.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:12.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:13.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:13.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:13.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:13.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:13.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:13.343+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19245 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:13.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:13.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:13.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:13.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:13.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:13.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:13.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:13.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:14.068+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:14.068+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:14.068+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:14.069+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:13.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:13.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:13.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:13.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:14.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:14.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:14.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:14.463+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:15.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:15.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:15.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:15.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:14.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:14.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:14.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:14.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:15.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:15.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:15.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:15.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:15.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:16.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:16.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:16.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:16.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:16.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:15.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:16.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:15.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:16.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:15.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:16.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:15.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:16.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:16.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:16.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:16.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:16.479+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:17.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:17.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:17.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:17.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:17.014+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:17.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:17.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:17.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:17.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:17.025+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:17.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:17.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:17.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:17.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:17.491+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:18.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:18.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:18.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:18.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:18.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:18.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:18.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:18.031+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:18.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:18.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:18.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:18.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:18.345+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19335 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:18.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:18.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:18.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:18.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:18.539+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:19.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:19.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:19.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:19.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:19.046+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:18.996+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:19.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:19.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:19.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:19.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:19.557+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:19.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:19.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:19.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:20.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:19.977+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.005+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:20.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:20.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:20.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:20.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:20.587+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:20.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:20.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:20.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:21.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:20.939+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:21.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:20.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:21.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:21.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:21.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:21.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:21.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:21.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:21.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:21.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:21.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:21.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:22.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:21.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:21.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:21.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:21.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:21.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:21.960+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:22.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:22.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:22.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:22.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:22.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:22.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:22.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:22.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:22.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:22.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:23.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:22.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:23.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:23.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:23.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:23.002+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:23.666 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:23.347+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19424 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:23.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:23.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:23.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:23.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:23.667+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:23.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:24.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:24.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:24.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:24.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:24.043+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:24.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:24.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:24.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:24.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:24.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:24.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:24.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:24.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:25.041+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:25.041+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:25.041+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:25.042+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:24.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:24.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:24.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:24.894+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:25.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:25.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:25.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:25.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:25.657+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:25.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:25.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:25.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:25.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:26.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:26.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:26.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:26.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:26.001+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:26.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:26.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:26.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:26.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:26.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:26.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:26.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:26.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:26.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:26.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:26.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:27.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:26.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:27.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:27.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:27.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:27.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:27.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:27.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:27.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:27.660+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:27.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:27.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:27.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:28.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:27.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:28.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:28.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:28.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:28.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:28.006+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:28.655 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:28.349+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19514 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:28.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:28.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:28.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:28.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:28.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:28.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:28.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:28.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:28.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:29.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.018+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:29.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:29.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:29.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:29.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:29.648+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:29.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:29.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:29.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:29.822+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:30.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:29.989+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:30.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:30.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:30.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:30.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:30.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:30.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:30.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:30.616+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:30.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:30.782+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:30.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:30.782+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:30.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:30.782+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:30.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:30.783+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:31.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:31.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:31.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:31.021+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:31.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:31.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:31.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:31.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:31.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:31.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:31.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:31.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:31.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:31.805+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:32.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:32.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:32.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:32.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:32.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:32.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:32.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:32.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:32.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:32.606+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:32.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:32.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:32.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:32.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:32.777+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:33.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:33.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:33.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:33.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:33.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:33.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:33.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:33.044+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:33.638 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:33.351+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19604 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:33.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:33.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:33.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:33.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:33.810+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:33.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:33.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:33.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:33.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:33.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:34.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:34.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:34.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:34.049+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:34.855+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:34.855+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:34.855+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:34.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:34.856+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:34.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:34.612+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:34.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:34.612+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:34.612+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:34.613+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:35.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:35.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:35.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:35.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:35.075+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:35.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:35.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:35.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:35.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:35.859+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:35.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:35.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:35.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:35.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:35.660+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:36.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:36.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:36.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:36.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:36.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:36.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:36.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:36.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:36.628+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:36.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:36.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:36.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:36.816+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:37.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:37.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:37.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:37.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:37.005+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:37.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:37.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:37.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:37.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:37.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:37.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:37.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:37.672+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:38.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:37.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:37.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:37.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:37.769+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:38.658 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:38.353+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19694 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:38.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:38.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:38.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:38.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:38.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:38.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:38.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:38.659+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:38.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:38.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:38.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:38.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:39.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:38.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:39.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:39.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:39.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:39.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:39.625+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:39.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:39.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:39.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:39.753+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:39.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:39.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:39.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:39.977+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:40.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:40.720+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:40.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:40.720+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:40.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:40.720+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:40.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:40.721+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:40.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:40.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:40.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:40.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:40.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:40.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:40.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:40.661+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:41.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:40.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:40.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:40.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:40.937+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:41.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:41.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:41.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:41.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:41.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:41.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:41.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:41.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:41.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:41.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:42.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:41.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:42.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:41.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:42.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:41.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:42.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:41.945+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:42.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:42.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:42.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:42.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:42.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:42.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:42.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:42.599+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:42.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:42.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:42.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:42.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:42.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:42.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:42.902 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:42.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:42.900+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:42.900+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:42.900+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:42.901+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:43.479 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:43.355+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19783 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:43.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:43.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:43.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:43.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:43.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:43.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:43.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:43.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:43.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:43.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:44.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:43.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:44.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:43.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:44.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:43.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:44.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:43.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:44.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:44.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:44.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:44.661+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:44.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:44.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:44.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:44.614+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:44.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:44.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:44.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:44.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:45.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:45.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:45.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:45.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:45.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:45.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:45.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:45.640+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:45.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:45.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:45.894 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:45.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:45.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:45.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:45.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:45.620+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:46.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:45.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:45.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:45.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:46.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:45.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:46.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:46.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:46.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:46.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:46.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:46.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:46.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:46.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:46.636+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:46.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:46.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:46.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:46.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:47.854 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:47.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:47.854 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:47.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:47.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:47.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:47.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:47.590+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:47.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:47.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:47.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:47.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:47.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:47.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:47.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:47.609+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:47.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:47.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:47.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:48.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:47.855+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:48.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:48.357+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19873 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:48.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:48.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:48.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:48.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:48.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:48.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:48.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:48.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:48.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:48.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:48.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:48.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:48.877+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:48.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:48.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:48.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:48.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:48.653+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:49.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:49.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:49.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:49.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:49.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:49.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:49.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:49.523+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:49.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:49.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:49.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:49.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:49.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:49.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:49.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:49.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:49.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:50.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:49.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:49.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:49.927+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:50.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:50.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:50.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:50.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:50.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:50.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:50.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:50.488+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:50.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:50.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:50.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:50.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:50.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:50.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:50.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:50.616+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:50.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:50.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:50.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:50.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:51.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:51.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:51.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:51.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:51.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:51.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:51.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:51.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:51.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:51.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:51.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:51.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:51.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:51.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:51.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:51.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:51.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:51.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:51.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:51.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:51.915+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:52.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:52.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:52.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:52.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:52.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:52.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:52.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:52.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:52.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:52.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:53.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:52.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:52.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:52.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:52.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:53.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:53.359+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19962 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:53.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:53.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:53.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:53.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:53.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:53.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:53.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:53.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:53.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:53.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:53.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:53.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:53.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:53.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:54.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:53.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:54.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:54.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:54.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:54.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:54.833 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:54.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:54.833 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:54.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:54.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:54.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:54.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:54.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:54.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:54.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:54.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:54.591+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:55.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:54.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:54.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:54.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:55.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:54.833+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:55.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:55.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:56.269 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:55.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:56.269 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:55.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:56.269 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:55.577+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:56.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:55.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:56.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:55.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:56.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:55.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:56.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:55.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:56.271 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:55.812+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:56.271 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:55.812+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:56.271 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:55.812+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:56.271 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:55.813+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:56.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:56.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:56.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:56.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:56.424+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:56.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:56.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:56.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:56.612+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:56.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:56.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:56.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:57.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:56.769+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:57.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:57.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:57.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:57.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:57.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:57.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:57.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:57.386+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:57.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:57.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:57.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:57.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:57.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:57.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:57.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:57.638+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:58.658 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:21:58.361+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20051 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:21:58.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:58.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:58.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:58.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:58.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:58.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:58.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:58.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:58.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:58.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:58.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:21:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:58.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:58.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:58.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:58.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:58.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:58.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:59.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:59.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:59.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:21:59.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:59.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:59.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:59.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:21:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:21:59.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:00.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:59.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:00.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:59.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:00.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:59.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:00.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:21:59.824+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:00.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:00.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:00.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:00.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:00.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:00.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:00.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:00.479+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:00.847 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:00.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:00.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:00.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:00.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:00.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:00.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:00.595+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:00.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:00.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:00.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:01.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:00.845+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:01.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:01.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:01.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:01.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:01.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:01.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:01.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:01.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:01.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:01.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:01.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:01.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:01.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:01.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:01.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:01.444+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:01.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:01.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:01.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:01.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:02.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:02.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:02.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:02.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:02.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:02.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:02.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:02.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:02.651+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:03.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:02.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:03.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:02.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:03.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:02.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:03.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:02.819+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:03.680 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:03.363+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20140 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:03.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:03.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:03.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:03.432+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:03.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:03.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:03.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:03.812+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:03.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:03.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:03.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:03.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:03.681+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:04.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:04.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:04.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:04.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:04.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:04.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:04.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:04.665+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:04.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:04.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:04.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:04.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:04.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:04.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:04.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:04.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:05.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:04.856+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:05.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:05.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:05.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:05.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:05.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:05.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:05.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:05.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:05.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:05.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:05.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:05.849+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:05.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:05.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:05.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:05.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:06.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:06.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:06.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:06.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:06.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:06.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:06.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:06.431+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:06.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:06.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:06.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:06.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:06.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:06.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:06.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:06.859+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:06.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:06.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:06.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:06.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:06.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:06.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:07.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:07.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:07.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:07.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:07.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:07.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:07.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:07.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:07.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:07.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:07.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:07.833+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:07.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:07.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:07.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:07.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:07.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:08.678 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:08.365+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20230 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:08.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:08.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:08.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:08.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:08.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:08.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:08.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:08.432+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:08.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:08.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:08.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:08.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:08.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:08.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:08.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:08.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:08.818+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:09.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:09.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:09.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:09.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:09.669+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:09.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:09.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:09.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:09.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:10.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:09.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:10.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:09.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:10.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:09.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:10.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:09.794+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:10.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:10.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:10.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:10.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:10.437+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:10.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:10.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:10.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:10.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:10.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:10.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:10.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:10.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:11.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:10.817+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:11.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:11.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:11.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:11.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:11.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:11.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:11.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:11.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:11.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:11.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:11.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:11.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:11.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:11.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:11.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:11.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:11.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:11.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:11.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:11.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:11.792+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:12.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:12.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:12.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:12.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:12.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:12.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:12.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:12.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:12.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:12.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:12.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:12.633+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:12.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:12.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:12.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:13.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:12.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:13.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:13.367+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20320 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:13.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:13.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:13.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:13.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:13.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:13.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:13.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:13.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:13.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:13.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:13.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:13.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:13.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:13.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:13.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:13.781+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:13.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:13.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:13.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:13.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:13.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:13.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:13.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:13.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:14.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:14.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:14.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:14.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:14.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:14.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:14.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:14.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:14.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:15.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:14.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:15.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:14.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:15.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:14.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:15.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:14.802+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:15.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:15.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:15.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:15.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:15.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:15.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:15.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:15.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:15.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:15.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:15.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:15.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:15.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:16.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:15.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:15.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:15.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:16.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:15.819+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:16.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:16.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:16.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:16.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:16.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:16.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:16.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:16.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:16.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:16.562+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:16.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:17.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:16.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:16.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:16.813+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:17.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:17.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:17.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:17.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:17.391+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:17.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:17.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:17.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:17.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:17.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:17.541+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:17.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:17.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:17.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:18.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:17.796+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:18.369+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20410 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:18.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:18.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:18.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:18.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:18.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:18.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:18.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:18.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:18.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:18.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:18.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:18.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:18.778+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:19.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:19.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:19.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:19.370+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:19.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:19.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:19.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:19.524+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:20.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:19.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:20.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:19.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:20.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:19.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:20.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:19.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:20.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:20.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:20.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:20.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:20.771+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:20.771+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:20.771+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:20.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:20.771+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:20.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:20.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:20.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:20.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:20.528+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:21.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:21.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:21.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:21.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:21.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:21.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:21.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:21.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:21.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:21.366+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:21.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:21.366+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:21.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:21.366+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:21.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:21.366+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:21.799+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:21.799+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:21.799+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:21.800+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:22.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:22.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:22.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:22.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:22.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:22.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:22.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:22.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:22.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:22.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:22.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:22.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:22.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:22.476+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:23.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:22.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:23.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:22.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:23.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:22.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:23.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:22.812+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:23.464 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:23.371+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20496 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:23.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:23.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:23.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:23.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:23.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:23.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:23.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:23.370+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:23.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:23.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:23.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:23.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:23.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:23.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:23.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:23.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:24.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:23.794+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:24.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:24.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:24.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:24.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:24.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:24.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:24.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:24.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:24.450+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:24.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:24.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:24.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:24.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:25.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:25.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:25.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:25.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:25.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:25.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:25.456+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:25.456+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:25.456+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:25.457+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:25.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:25.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:25.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:26.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:25.779+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:26.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:26.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:26.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:26.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:26.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:26.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:26.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:26.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:26.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:26.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:26.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:26.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:26.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:26.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:26.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:26.497+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:27.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:26.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:27.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:26.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:27.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:26.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:27.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:26.827+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:27.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:27.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:27.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:27.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:27.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:27.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:27.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:27.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:27.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:27.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:27.456+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:28.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:27.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:28.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:27.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:28.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:27.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:28.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:27.821+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:28.373+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20580 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:28.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:28.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:28.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:28.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:28.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:28.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:28.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:28.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:28.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:28.439+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:29.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:28.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:29.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:28.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:29.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:28.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:29.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:28.789+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:29.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:29.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:29.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:29.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:29.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:29.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:29.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:29.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:29.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:29.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:29.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:29.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:30.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:29.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:30.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:29.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:30.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:29.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:30.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:29.837+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:30.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:30.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:30.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:30.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:30.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:30.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:30.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:30.357+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:30.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:30.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:30.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:30.434+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:30.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:30.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:30.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:30.836+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:31.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:31.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:31.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:31.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:31.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:31.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:31.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:31.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:31.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:31.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:31.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:31.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:31.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:31.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:31.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:31.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:31.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:31.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:31.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:31.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:31.845+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:32.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:32.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:32.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:32.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:32.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:32.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:32.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:32.363+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:32.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:32.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:32.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:32.412+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:33.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:32.836+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:33.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:32.836+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:33.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:32.836+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:33.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:32.836+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:33.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:33.375+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20660 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:33.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:33.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:33.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:33.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:33.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:33.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:33.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:33.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:33.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:33.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:33.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:33.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:34.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:33.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:34.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:33.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:34.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:33.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:34.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:33.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:34.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:34.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:34.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:34.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:34.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:34.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:34.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:34.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:34.396+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:34.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:34.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:34.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:34.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:35.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:35.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:35.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:35.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:35.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:35.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:35.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:35.302+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:35.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:35.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:35.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:35.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:35.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:35.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:35.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:35.822+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:36.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:36.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:36.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:36.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:36.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:36.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:36.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:36.355+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:36.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:36.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:36.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:36.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:37.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:36.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:37.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:36.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:37.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:36.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:37.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:36.804+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:37.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:37.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:37.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:37.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:37.328+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:37.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:37.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:37.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:37.317+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:37.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:37.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:37.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:37.812+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:38.378+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20741 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:38.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:38.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:38.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:38.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:38.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:38.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:38.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:38.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:38.272+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:39.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:38.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:39.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:38.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:39.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:38.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:39.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:38.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:39.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:39.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:39.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:39.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:39.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:39.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:39.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:39.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:39.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:39.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:39.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:39.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:40.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:39.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:40.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:39.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:40.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:39.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:40.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:39.749+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:40.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:40.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:40.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:40.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:40.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:40.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:40.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:40.363+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:40.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:40.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:40.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:41.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:40.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:41.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:41.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:41.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:41.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:41.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:41.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:41.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:41.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:41.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:41.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:41.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:41.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:41.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:41.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:41.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:41.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:41.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:41.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:41.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:41.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:41.768+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:42.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:42.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:42.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:42.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:42.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:42.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:42.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:42.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:42.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:42.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:42.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:42.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:43.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:42.742+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:43.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:43.380+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20821 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:43.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:43.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:43.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:43.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:43.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:43.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:43.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:43.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:43.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:43.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:43.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:43.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:43.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:43.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:43.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:43.256+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:43.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:43.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:43.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:43.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:43.702+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:44.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:44.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:44.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:44.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:44.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:44.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:44.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:44.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:44.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:44.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:44.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:44.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:44.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:44.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:44.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:44.427+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:44.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:44.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:44.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:44.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:44.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:45.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:45.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:45.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:45.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:45.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:45.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:45.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:45.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:45.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:45.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:45.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:45.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:45.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:45.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:45.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:45.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:45.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:45.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:45.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:45.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:45.711+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:46.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:46.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:46.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:46.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:46.243+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:46.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:46.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:46.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:46.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:47.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:46.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:47.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:46.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:47.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:46.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:47.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:46.742+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:47.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:47.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:47.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:47.200+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:47.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:47.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:47.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:47.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:47.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:47.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:47.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:47.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:47.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:47.721+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:48.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:48.382+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20907 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:48.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:48.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:48.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:48.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:48.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:48.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:48.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:48.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:48.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:49.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:48.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:49.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:48.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:49.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:48.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:49.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:48.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:49.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:49.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:49.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:49.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:49.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:49.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:49.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:49.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:49.217+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:50.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:49.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:50.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:49.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:50.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:49.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:50.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:49.727+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:50.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:50.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:50.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:50.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:50.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:50.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:50.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:50.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:50.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:50.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:50.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:50.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:50.698+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:51.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:51.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:51.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:51.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:51.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:51.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:51.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:51.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:51.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:51.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:51.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:51.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:51.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:51.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:51.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:51.407+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:51.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:51.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:51.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:51.727+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:52.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:52.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:52.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:52.184+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:52.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:52.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:52.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:52.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:52.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:52.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:52.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:53.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:52.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:53.509 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:53.384+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20994 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:53.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:53.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:53.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:53.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:53.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:53.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:53.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:53.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:53.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:53.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:53.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:53.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:53.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:53.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:53.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:53.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:53.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:53.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:53.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:53.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:53.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:53.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:53.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:53.710+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:54.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:54.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:54.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:54.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:54.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:54.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:54.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:54.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:54.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:54.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:54.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:54.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:55.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:55.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:55.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:55.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:55.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:55.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:55.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:55.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:55.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:55.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:55.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:55.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:55.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:55.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:55.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:55.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:55.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:55.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:55.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:55.666+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:56.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:56.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:56.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:56.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:56.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:56.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:56.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:56.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:56.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:56.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:56.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:56.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:56.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:56.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:56.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:56.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:56.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:56.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:56.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:56.628+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:57.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:57.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:57.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:57.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:57.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:57.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:57.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:57.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:57.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:57.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:57.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:57.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:57.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:57.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:57.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:57.387+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:57.655+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:57.655+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:57.655+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:57.656+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:58.648 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:22:58.386+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21084 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:22:58.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:58.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:58.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:58.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:58.315+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:58.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:58.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:58.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:58.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:58.373+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:58.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:58.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:58.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:58.648+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:59.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:59.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:59.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:59.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:59.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:59.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:22:59.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:22:59.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:59.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:59.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:59.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:59.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:59.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:59.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:22:59.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:22:59.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:22:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:59.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:59.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:59.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:22:59.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:22:59.608+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:00.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:00.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:00.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:00.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:00.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:00.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:00.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:00.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:00.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:00.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:00.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:00.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:00.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:00.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:00.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:00.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:00.427+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:01.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:01.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:01.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:01.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:01.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:01.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:01.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:01.232+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:01.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:01.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:01.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:01.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:01.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:01.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:01.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:01.468+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:01.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:01.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:01.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:01.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:01.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:02.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:02.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:02.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:02.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:02.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:02.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:02.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:02.496+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:02.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:02.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:02.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:02.256+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:02.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:02.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:02.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:02.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:02.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:03.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:03.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:03.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:03.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:03.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:03.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:03.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:03.211+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:03.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:03.388+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21171 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:03.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:03.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:03.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:03.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:03.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:03.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:03.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:03.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:03.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:04.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:04.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:04.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:04.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:04.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:04.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:04.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:04.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:04.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:04.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:04.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:04.525+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:04.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:04.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:04.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:04.473+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:05.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:05.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:05.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:05.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:05.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:05.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:05.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:05.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:05.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:05.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:05.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:05.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:05.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:05.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:05.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:05.451+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:05.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:05.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:05.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:05.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:05.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:06.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:06.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:06.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:06.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:06.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:06.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:06.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:06.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:06.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:06.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:06.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:06.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:06.556+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:06.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:06.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:06.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:06.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:07.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:07.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:07.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:07.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:07.433+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:07.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:07.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:07.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:07.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:07.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:07.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:07.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:07.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:08.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:08.391+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21259 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:08.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:08.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:08.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:08.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:08.182+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:08.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:08.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:08.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:08.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:08.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:08.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:08.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:08.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:08.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:09.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:09.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:09.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:09.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:09.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:09.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:09.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:09.226+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:09.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:09.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:09.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:09.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:09.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:09.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:09.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:09.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:09.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:09.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:09.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:09.513+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:10.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:10.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:10.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:10.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:10.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:10.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:10.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:10.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:10.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:10.190+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:10.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:10.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:10.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:10.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:10.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:11.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:11.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:11.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:11.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:11.379+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:11.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:11.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:11.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:11.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:11.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:11.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:11.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:11.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:12.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:12.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:12.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:12.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:12.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:12.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:12.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:12.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:12.147+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:12.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:12.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:12.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:12.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:13.470 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:13.393+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21348 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:13.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:13.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:13.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:13.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:13.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:13.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:13.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:13.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:13.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:13.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:13.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:13.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:13.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:13.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:13.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:13.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:14.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:14.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:14.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:14.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:14.349+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:14.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:14.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:14.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:14.212+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:14.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:14.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:14.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:14.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:14.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:15.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:15.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:15.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:15.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:15.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:15.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:15.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:15.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:15.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:15.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:15.559+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:15.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:16.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:16.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:16.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:16.167+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:16.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:16.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:16.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:16.560+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:16.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:16.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:16.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:16.423+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:17.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:17.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:17.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:17.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:17.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:17.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:17.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:17.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:17.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:17.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:17.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:17.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:17.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:17.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:17.559+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:18.521 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:18.395+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21438 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:18.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:18.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:18.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:18.362+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:18.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:18.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:18.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:18.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:18.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:18.223+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:18.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:18.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:18.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:18.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:18.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:18.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:19.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:19.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:19.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:19.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:19.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:19.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:19.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:19.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:19.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:19.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:19.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:19.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:19.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:19.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:19.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:19.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:20.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:20.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:20.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:20.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:20.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:20.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:20.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:20.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:20.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:20.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:20.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:20.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:20.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:20.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:20.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:20.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:20.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:20.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:21.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:21.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:21.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:21.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:21.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:21.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:21.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:21.181+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:21.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:21.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:21.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:21.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:21.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:21.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:21.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:21.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:21.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:21.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:21.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:21.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:21.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:21.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:21.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:21.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:22.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:22.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:22.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:22.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:22.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:22.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:22.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:22.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:22.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:22.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:22.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:22.499+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:22.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:22.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:22.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:22.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:23.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:23.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:23.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:23.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:23.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:23.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:23.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:23.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:23.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:23.397+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21527 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:23.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:23.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:23.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:23.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:23.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:23.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:23.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:23.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:23.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:23.440+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:24.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:24.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:24.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:24.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:24.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:24.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:24.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:24.476+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:24.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:24.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:24.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:24.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:25.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:25.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:25.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:25.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:25.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:25.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:25.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:25.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:25.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:25.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:25.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:25.484+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:26.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:26.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:26.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:26.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:26.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:26.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:26.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:26.190+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:26.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:26.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:26.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:26.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:26.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:26.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:26.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:26.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:26.462+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:27.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:27.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:27.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:27.149+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:27.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:27.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:27.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:27.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:27.457+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:27.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:27.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:27.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:27.493+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:28.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:28.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:28.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:28.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:28.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:28.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:28.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:28.135+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:28.399+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21617 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:28.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:28.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:28.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:28.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:28.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:28.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:28.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:28.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:28.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:29.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:29.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:29.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:29.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:29.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:29.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:29.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:29.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:29.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:29.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:29.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:29.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:29.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:29.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:30.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:30.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:30.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:30.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:30.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:30.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:30.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:30.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:30.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:30.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:30.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:30.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:30.487+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:31.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:31.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:31.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:31.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:31.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:31.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:31.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:31.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:31.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:31.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:31.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:31.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:31.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:31.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:31.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:31.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:31.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:31.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:31.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:31.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:31.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:31.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:31.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:31.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:32.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:32.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:32.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:32.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:32.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:32.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:32.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:32.066+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:32.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:32.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:32.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:32.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:32.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:32.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:32.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:32.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:32.556+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:33.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:33.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:33.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:33.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:33.103+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:33.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:33.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:33.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:33.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:33.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:33.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:33.401+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21706 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:33.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:33.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:33.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:33.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:33.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:33.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:33.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:33.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:34.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:34.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:34.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:34.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:34.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:34.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:34.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:34.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:34.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:34.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:34.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:34.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:34.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:34.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:34.338+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:35.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:35.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:35.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:35.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:35.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:35.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:35.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:35.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:35.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:35.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:35.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:35.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:35.545+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:35.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:35.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:35.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:35.361+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:36.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:36.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:36.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:36.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:36.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:36.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:36.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:36.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:36.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:36.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:36.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:36.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:36.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:36.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:36.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:36.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:36.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:37.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:37.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:37.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:37.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:37.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:37.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:37.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:37.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:37.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:37.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:37.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:37.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:37.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:37.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:37.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:37.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:37.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:38.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:38.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:38.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:38.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:38.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:38.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:38.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:38.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:38.404+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21796 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:38.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:38.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:38.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:38.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:38.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:38.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:38.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:38.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:39.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:39.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:39.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:39.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:39.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:39.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:39.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:39.044+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:39.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:39.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:39.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:39.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:39.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:39.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:39.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:39.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:39.343+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:40.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:40.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:40.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:40.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:40.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:40.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:40.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:40.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:40.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:40.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:40.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:40.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:40.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:40.519+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:41.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:41.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:41.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:41.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:41.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:41.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:41.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:41.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:41.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:41.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:41.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:41.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:41.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:41.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:41.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:41.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:41.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:41.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:41.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:41.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:41.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:41.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:41.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:41.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:42.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:42.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:42.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:42.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:42.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:42.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:42.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:42.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:42.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:42.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:42.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:42.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:42.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:42.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:42.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:42.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:43.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:43.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:43.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:43.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:43.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:43.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:43.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:43.073+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:43.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:43.405+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21884 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:43.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:43.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:43.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:43.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:43.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:43.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:43.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:43.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:43.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:43.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:43.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:43.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:43.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:43.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:43.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:43.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:44.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:44.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:44.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:44.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:44.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:44.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:44.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:44.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:44.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:44.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:44.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:44.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:44.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:44.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:44.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:44.300+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:44.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:44.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:44.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:44.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:44.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:45.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:45.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:45.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:45.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:45.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:45.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:45.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:45.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:45.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:45.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:45.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:45.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:45.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:45.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:45.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:45.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:45.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:45.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:45.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:45.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:45.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:46.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:46.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:46.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:46.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:46.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:46.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:46.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:46.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:46.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:46.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:46.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:46.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:47.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:47.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:47.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:47.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:47.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:47.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:47.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:47.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:47.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:47.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:47.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:47.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:47.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:47.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:47.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:47.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:47.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:47.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:47.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:47.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:47.457+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:48.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:48.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:48.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:48.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:48.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:48.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:48.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:48.128+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:48.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:48.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:48.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:48.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:48.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:48.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:48.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:48.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:48.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:48.409+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21975 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:48.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:48.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:48.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:48.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:48.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:49.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:49.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:49.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:49.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:49.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:49.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:49.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:49.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:49.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:49.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:49.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:49.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:49.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:49.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:49.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:50.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:50.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:50.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:50.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:50.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:50.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:50.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:50.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:50.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:50.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:50.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:50.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:50.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:50.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:50.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:50.300+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:50.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:50.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:50.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:50.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:50.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:51.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:51.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:51.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:51.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:51.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:51.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:51.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:51.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:51.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:51.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:51.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:51.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:51.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:51.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:51.500+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:52.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:52.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:52.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:52.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:52.097+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:52.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:52.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:52.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:52.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:52.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:52.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:52.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:52.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:52.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:53.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:53.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:53.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:53.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:53.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:53.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:53.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:53.119+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:53.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:53.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:53.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:53.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:53.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:53.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:53.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:53.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:53.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:53.411+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22064 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:53.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:53.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:53.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:53.584+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:54.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:54.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:54.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:54.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:54.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:54.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:54.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:54.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:54.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:54.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:54.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:54.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:54.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:55.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:55.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:55.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:55.158+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:55.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:55.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:55.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:55.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:55.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:55.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:55.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:55.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:55.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:55.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:56.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:56.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:56.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:56.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:56.349+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:56.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:56.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:56.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:56.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:56.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:56.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:56.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:56.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:56.571+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:57.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:57.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:57.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:57.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:57.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:57.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:57.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:57.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:57.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:57.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:57.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:57.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:57.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:57.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:58.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:58.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:58.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:58.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:58.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:58.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:58.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:58.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:23:58.413+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22155 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:58.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:58.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:58.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:58.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:58.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:58.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:58.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:58.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:59.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:59.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:59.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:59.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:59.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:59.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:23:59.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:23:59.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:59.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:59.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:59.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:59.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:23:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:23:59.430+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:23:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:59.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:59.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:59.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:23:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:23:59.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:00.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:00.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:00.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:00.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:00.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:00.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:00.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:00.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:00.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:00.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:00.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:00.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:00.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:00.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:00.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:00.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:00.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:00.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:00.431+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:01.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:01.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:01.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:01.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:01.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:01.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:01.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:01.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:01.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:01.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:01.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:01.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:01.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:01.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:01.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:01.603+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:01.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:01.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:01.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:01.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:01.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:01.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:01.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:01.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:02.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:02.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:02.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:02.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:02.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:02.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:02.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:02.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:02.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:02.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:02.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:02.626+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:02.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:02.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:02.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:02.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:03.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:03.415+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22242 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:03.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:03.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:03.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:03.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:03.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:03.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:03.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:03.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:03.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:03.400+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:03.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:03.400+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:03.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:03.400+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:03.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:03.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:03.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:03.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:03.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:03.633+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:04.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:04.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:04.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:04.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:04.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:04.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:04.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:04.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:04.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:04.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:04.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:04.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:04.614+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:04.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:04.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:04.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:04.440+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:05.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:05.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:05.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:05.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:05.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:05.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:05.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:05.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:05.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:05.486+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:05.655+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:05.655+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:05.655+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:05.656+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:06.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:06.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:06.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:06.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:06.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:06.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:06.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:06.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:06.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:06.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:06.510+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:06.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:06.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:06.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:06.684+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:07.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:07.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:07.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:07.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:07.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:07.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:07.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:07.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:07.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:07.708+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:07.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:07.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:07.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:07.553+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:08.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:08.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:08.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:08.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:08.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:08.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:08.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:08.102+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:08.417+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22333 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:08.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:08.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:08.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:08.664+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:08.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:08.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:08.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:08.522+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:09.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:09.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:09.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:09.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:09.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:09.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:09.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:09.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:09.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:09.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:09.615+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:09.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:09.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:09.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:09.473+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:10.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:10.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:10.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:10.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:10.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:10.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:10.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:10.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:10.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:10.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:10.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:10.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:10.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:10.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:10.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:10.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:10.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:10.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:10.649+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:11.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:11.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:11.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:11.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:11.140+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:11.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:11.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:11.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:11.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:11.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:11.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:11.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:11.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:12.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:12.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:12.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:12.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:12.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:12.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:12.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:12.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:12.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:12.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:12.490+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:13.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:13.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:13.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:13.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:13.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:13.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:13.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:13.127+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:13.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:13.420+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22422 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:13.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:13.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:13.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:13.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:14.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:13.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:14.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:13.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:14.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:13.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:14.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:13.733+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:14.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:14.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:14.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:14.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:14.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:14.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:14.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:14.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:14.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:14.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:14.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:14.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:14.462+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:14.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:14.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:14.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:14.687+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:15.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:15.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:15.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:15.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:15.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:15.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:15.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:15.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:15.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:15.685+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:15.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:15.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:15.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:15.495+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:16.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:16.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:16.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:16.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:16.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:16.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:16.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:16.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:16.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:16.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:16.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:16.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:16.666+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:16.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:16.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:16.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:16.462+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:17.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:17.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:17.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:17.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:17.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:17.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:17.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:17.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:17.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:17.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:17.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:17.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:17.677+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:17.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:17.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:17.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:17.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:18.075+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:18.075+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:18.075+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:18.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:18.076+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:18.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:18.422+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22513 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:18.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:18.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:18.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:18.460+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:19.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:18.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:19.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:18.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:19.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:18.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:19.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:18.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:19.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:19.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:19.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:19.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:19.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:19.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:19.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:19.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:19.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:19.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:19.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:19.747 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:19.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:20.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:19.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:20.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:19.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:20.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:19.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:20.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:19.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:20.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:20.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:20.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:20.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:20.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:20.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:20.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:20.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:20.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:20.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:20.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:20.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:21.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:20.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:21.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:20.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:21.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:20.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:21.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:20.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:21.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:21.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:21.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:21.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:21.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:21.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:21.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:21.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:21.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:21.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:21.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:21.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:21.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:21.764+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:21.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:21.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:21.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:21.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:22.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:22.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:22.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:22.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:22.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:22.068+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:22.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:22.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:22.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:22.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:22.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:22.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:22.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:22.496+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:23.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:22.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:23.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:22.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:23.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:22.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:23.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:22.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:23.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:23.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:23.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:23.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:23.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:23.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:23.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:23.030+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:23.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:23.425+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22601 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:23.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:23.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:23.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:23.535+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:24.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:23.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:24.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:23.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:24.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:23.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:24.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:23.763+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:24.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:24.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:24.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:24.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:24.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:24.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:24.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:24.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:24.573+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:24.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:24.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:24.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:24.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:25.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:25.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:25.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:25.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:25.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:25.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:25.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:25.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:25.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:25.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:25.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:25.798+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:25.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:25.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:25.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:25.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:26.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:26.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:26.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:26.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:26.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:26.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:26.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:26.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:26.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:26.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:26.814+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:26.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:26.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:26.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:26.573+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:27.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:27.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:27.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:27.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:27.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:27.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:27.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:27.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:27.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:27.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:27.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:27.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:27.555+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:28.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:27.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:28.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:27.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:28.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:27.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:28.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:27.803+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:28.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:28.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:28.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:28.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:28.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:28.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:28.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:28.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:28.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:28.427+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22692 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:28.565+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:28.565+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:28.565+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:28.566+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:29.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:28.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:29.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:28.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:29.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:28.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:29.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:28.810+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:29.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:29.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:29.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:29.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:29.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:29.858 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:29.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:29.859 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:29.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:29.859 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:29.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:29.859 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:29.589+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:30.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:29.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:30.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:29.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:30.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:29.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:30.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:29.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:30.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:30.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:30.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:30.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:30.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:30.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:30.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:30.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:30.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:30.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:30.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:30.909 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:30.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:31.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:31.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:31.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:31.133+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:30.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:30.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:30.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:30.908+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:31.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:31.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:31.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:31.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:31.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:31.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:31.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:31.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:32.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:31.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:32.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:31.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:31.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:31.946+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:32.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:32.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:32.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:32.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:32.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:32.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:32.926 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:32.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:32.926 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:32.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:32.926 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:32.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:33.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:33.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:33.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:33.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:33.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:32.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:32.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:32.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:32.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:33.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:33.428+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22781 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:33.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:33.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:33.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:33.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:34.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:33.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:34.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:33.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:34.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:33.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:34.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:33.878+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:34.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:34.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:34.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:34.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:34.135+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:34.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:34.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:34.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:34.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:34.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:34.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:34.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:34.645+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:34.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:34.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:34.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:34.922+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:35.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:35.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:35.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:35.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:35.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:35.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:35.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:35.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:35.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:35.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:35.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:35.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:35.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:36.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:35.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:36.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:35.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:36.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:35.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:36.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:35.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:36.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:36.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:36.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:36.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:36.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:36.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:36.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:36.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:36.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:36.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:36.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:36.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:36.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:36.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:37.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:37.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:37.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:37.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:37.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:36.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:36.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:36.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:36.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:37.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:37.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:37.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:37.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:37.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:37.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:37.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:37.684+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:38.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:37.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:38.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:37.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:38.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:37.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:38.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:37.970+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:38.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:38.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:38.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:38.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:38.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:38.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:38.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:38.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:38.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:38.432+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22872 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:38.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:38.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:38.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:38.681+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:39.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:39.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:39.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:39.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:39.125+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:39.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:39.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:39.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:39.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:39.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:39.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:39.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:39.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:39.704+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:40.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:40.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:40.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:40.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:40.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:41.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:41.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:41.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:41.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:40.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:41.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:40.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:41.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:40.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:41.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:40.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:41.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:40.731+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:41.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:41.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:41.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:41.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:41.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:41.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:41.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:41.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:41.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:41.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:41.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:41.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:42.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:42.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:42.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:42.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:42.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:41.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:41.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:41.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:41.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:42.996 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:42.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:42.997 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:42.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:42.997 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:42.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:42.997 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:42.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:43.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:43.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:43.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:43.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:43.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:43.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:43.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:43.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:43.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:42.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:43.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:42.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:43.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:42.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:43.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:42.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:43.717 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:43.434+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22960 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:43.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:43.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:43.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:43.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:43.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:43.717+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:44.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:44.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:44.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:44.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:44.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:43.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:43.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:43.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:43.976+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:44.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:44.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:44.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:44.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:44.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:44.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:44.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:44.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:45.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:45.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:45.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:45.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:44.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:44.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:44.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:44.951+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:45.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:45.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:45.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:45.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:45.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:46.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:46.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:46.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:46.010+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:45.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:45.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:45.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:45.980+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:46.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:46.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:46.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:46.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:46.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:47.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:47.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:47.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:47.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:47.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:46.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:46.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:46.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:46.977+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:47.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:47.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:47.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:47.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:47.679+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:48.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:48.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:48.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:48.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:48.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:48.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:48.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:48.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:48.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:48.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:48.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:48.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:48.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:48.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:48.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:48.007+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:48.697 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:48.436+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23051 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:48.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:48.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:48.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:48.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:48.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:48.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:48.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:48.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:49.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:49.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:49.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:49.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:49.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:49.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:49.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:49.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:49.014+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:49.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:49.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:49.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:49.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:50.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:50.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:50.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:50.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:50.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.016+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:50.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:50.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:50.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:50.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:50.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:50.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:50.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:50.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:51.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.967+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.967+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.967+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:50.968+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:51.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:51.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:51.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:51.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:51.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:51.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:51.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:51.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:51.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:51.632+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:52.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:51.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:51.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:51.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:51.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:52.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:52.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:52.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:52.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:52.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:52.014+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:52.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:52.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:52.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:52.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:53.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:53.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:53.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:53.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:53.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:52.999+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:52.999+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:52.999+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:53.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:52.999+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:53.694 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:53.439+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23140 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:53.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:53.672+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:53.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:53.672+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:53.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:53.672+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:53.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:53.673+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:54.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:54.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:54.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:54.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:53.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:53.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:53.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:53.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:54.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:54.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:54.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:54.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:54.637+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:55.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:55.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:55.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:55.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:55.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:55.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:55.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:55.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:55.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:55.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:55.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:55.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:55.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:55.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:56.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:56.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:56.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:56.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:56.056+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:56.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:56.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:56.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:56.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:56.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:56.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:56.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:56.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:56.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:57.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:57.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:57.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:57.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:57.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.013+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:57.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:57.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:57.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:57.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:57.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:58.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:58.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:58.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:58.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:57.971+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:58.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:58.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:58.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:58.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:58.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:58.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:58.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:58.034+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:58.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:24:58.441+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23231 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:24:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:58.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:58.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:58.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:58.665+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:58.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:58.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:58.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:24:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:58.955+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:24:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:59.050+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:59.050+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:59.050+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:24:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:24:59.051+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:59.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:59.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:59.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:24:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:24:59.712+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:00.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:00.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:00.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:00.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:59.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:59.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:59.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:24:59.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:01.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:00.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:01.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:00.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:01.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:00.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:01.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:00.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:01.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:01.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:01.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:01.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:01.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:01.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:01.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:01.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:01.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:01.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:01.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:01.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:01.781+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:02.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:02.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:02.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:02.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:02.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:02.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:02.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:02.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:03.054 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:02.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:03.054 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:02.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:03.054 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:02.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:03.054 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:02.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:03.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:03.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:03.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:03.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:03.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:03.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:03.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:03.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:03.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:03.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:03.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:03.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:03.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:03.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:03.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:03.177+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:03.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:03.444+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23319 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:04.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:04.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:04.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:04.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:04.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:04.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:04.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:04.010+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:04.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:03.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:04.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:03.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:04.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:03.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:04.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:03.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:04.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:04.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:04.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:04.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:04.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:04.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:04.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:04.128+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:05.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:05.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:05.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:05.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.018+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:05.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:04.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:05.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:04.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:05.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:04.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:05.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:04.865+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:05.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:05.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:05.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:05.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:05.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:06.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:06.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:06.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:06.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:06.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:05.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:05.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:05.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:05.887+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:05.984+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:07.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:07.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:07.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:07.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.010+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:07.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:06.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:06.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:06.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:06.843+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:07.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:07.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:07.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:07.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:07.122+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:08.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:08.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:08.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:08.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:07.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:08.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:07.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:08.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:07.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:08.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:07.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:08.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:07.867+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:08.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:08.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:08.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:08.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:08.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:08.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:08.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:08.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:08.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:08.446+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23410 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:09.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:08.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:09.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:08.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:09.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:08.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:09.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:08.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:09.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:08.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:09.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:08.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:09.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:08.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:09.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:08.823+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:09.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:09.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:09.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:09.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:09.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:09.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:09.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:09.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:10.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:10.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:10.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:10.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.002+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:10.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:09.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:10.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:09.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:10.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:09.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:10.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:09.782+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:10.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:10.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:10.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:10.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:10.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:11.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:11.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:11.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:11.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:10.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:11.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:10.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:11.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:10.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:11.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:10.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:11.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:10.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:11.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:11.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:11.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:11.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:11.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:11.174+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:11.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:11.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:11.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:11.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:11.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:11.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:11.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:11.732+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:11.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:11.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:11.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:11.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:12.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:12.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:12.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:12.203+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:13.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:12.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:13.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:12.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:13.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:12.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:13.003 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:12.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:13.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:13.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:13.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:13.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.002+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:13.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:13.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:13.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:13.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:13.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:13.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:13.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:13.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:13.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:13.449+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23499 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:13.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:13.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:13.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:13.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:13.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:13.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:13.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:13.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:13.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:13.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:13.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:13.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:13.733+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:14.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:14.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:14.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:14.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:14.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:14.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:14.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:14.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:14.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:14.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:14.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:14.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:14.938+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:14.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:14.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:14.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:14.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:15.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:15.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:15.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:15.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:15.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:15.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:15.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:15.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:15.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:15.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:15.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:15.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:15.672+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:15.947+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:15.947+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:15.947+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:15.948+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:16.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:16.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:16.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:16.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:16.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:16.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:16.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:16.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:16.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:16.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:16.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:16.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:16.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:17.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:16.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:17.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:16.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:17.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:16.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:17.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:16.984+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:17.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:17.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:17.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:17.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:17.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:17.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:17.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:17.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:17.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:17.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:17.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:17.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:17.736+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:18.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:18.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:18.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:18.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:18.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:18.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:18.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:18.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:18.574 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:18.451+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23590 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:18.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:18.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:18.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:18.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:18.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:18.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:18.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:18.271+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:18.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:18.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:18.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:18.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:18.763+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:19.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:19.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:19.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:19.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:19.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:19.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:19.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:19.034+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:19.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:19.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:19.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:19.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:19.315+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:20.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:19.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:20.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:19.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:20.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:19.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:20.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:19.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:20.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:20.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:20.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:20.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:20.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:20.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:20.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:20.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:20.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:20.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:20.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:20.324+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:21.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:20.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:21.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:20.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:21.074 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:20.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:21.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:20.732+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:21.342 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:21.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:21.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:21.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:21.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:21.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:21.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:21.322+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:21.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:21.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:21.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:21.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:21.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:21.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:21.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:21.083+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:21.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:21.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:21.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:21.708+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:22.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:22.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:22.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:22.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:22.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:22.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:22.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:22.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:22.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:22.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:22.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:22.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:23.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:22.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:23.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:22.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:23.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:22.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:23.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:22.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:23.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:23.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:23.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:23.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:23.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:23.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:23.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:23.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:23.703 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:23.454+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23687 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:23.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:23.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:23.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:23.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:23.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:23.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:23.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:23.364+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:23.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:23.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:23.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:23.701+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:24.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:24.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:24.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:24.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.001+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:24.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:24.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:24.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:24.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:24.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:24.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:24.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:24.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:24.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:24.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:24.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:24.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:24.675+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:25.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:24.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:25.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:25.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:25.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:25.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:25.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:25.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:25.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:25.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:25.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:25.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:25.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:25.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:25.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:25.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:25.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:25.659+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:25.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:25.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:25.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:25.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:26.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:26.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:26.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:26.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:26.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:26.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:26.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:26.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:26.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:26.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:26.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:26.695+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:27.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:27.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:27.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:27.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:27.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:27.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:27.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:27.015+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:27.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:27.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:27.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:27.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:27.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:27.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:27.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:27.336+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:27.714+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:27.714+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:27.714+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:27.715+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:28.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:28.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:28.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:28.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.000+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:28.677 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:28.457+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23778 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:28.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:28.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:28.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:28.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:28.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:28.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:28.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:28.336+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:28.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:28.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:28.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:28.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:29.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:29.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:29.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:29.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:28.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:29.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:29.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:29.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:29.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:29.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:29.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:29.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:29.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:29.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:29.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:29.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:29.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:29.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:29.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:29.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:29.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:29.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:29.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:29.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:29.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:30.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:30.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:30.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:30.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:30.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:30.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:30.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:30.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:30.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:30.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:30.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:30.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:30.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:30.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:30.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:30.690+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:30.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:30.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:30.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:30.961+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:31.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:31.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:31.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:31.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:31.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:31.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:31.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:31.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:31.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:31.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:31.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:31.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:32.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:31.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:31.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:31.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:31.960+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:32.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:32.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:32.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:32.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:32.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:32.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:32.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:32.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:32.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:32.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:32.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:32.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:33.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:33.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:33.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:33.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:33.586 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:33.459+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23867 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:33.586 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:33.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:33.586 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:33.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:33.586 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:33.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:33.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:33.306+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:33.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:33.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:33.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:33.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:34.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:34.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:34.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:34.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:34.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:34.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:34.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:34.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:34.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:34.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:34.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:34.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:34.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:34.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:34.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:34.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:34.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:34.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:34.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:34.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:35.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:35.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:35.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:35.288+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:35.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:35.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:35.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:35.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:35.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:35.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:35.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:35.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:36.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:36.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:36.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:36.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:36.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:36.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:36.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:36.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:36.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:36.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:36.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:36.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:37.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:37.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:37.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:37.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:37.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:37.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:37.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:37.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:37.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:37.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:37.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:37.295+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:37.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:37.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:37.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:37.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:37.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:38.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:38.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:38.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:38.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:37.976+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:38.586 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:38.462+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23958 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:38.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:38.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:38.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:38.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:38.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:38.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:38.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:38.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:38.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:38.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:38.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:38.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:38.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:39.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:38.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:39.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:38.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:39.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:38.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:39.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:38.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:39.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:39.275+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:39.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:39.275+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:39.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:39.275+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:39.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:39.276+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:39.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:39.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:39.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:39.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:39.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:39.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:39.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:39.646+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:40.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:39.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:40.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:39.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:40.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:39.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:40.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:39.935+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:40.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:40.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:40.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:40.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:40.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:40.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:40.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:40.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:40.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:40.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:40.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:40.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:40.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:41.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:40.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:41.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:40.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:41.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:40.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:41.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:40.933+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:41.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:41.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:41.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:41.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:41.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:41.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:41.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:41.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:41.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:41.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:41.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:41.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:41.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:41.611+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:42.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:41.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:41.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:41.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:41.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:42.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:42.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:42.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:42.191+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:42.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:42.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:42.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:42.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:42.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:42.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:42.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:42.602+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:43.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:42.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:43.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:42.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:43.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:42.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:43.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:42.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:43.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:43.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:43.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:43.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:43.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:43.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:43.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:43.193+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:43.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:43.464+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24046 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:43.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:43.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:43.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:43.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:43.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:43.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:43.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:43.594+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:44.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:44.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:44.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:44.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:44.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:44.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:44.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:44.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:44.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:44.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:44.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:44.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:44.575+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:45.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:45.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:45.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:45.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:45.200+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:44.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:45.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:45.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:45.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:45.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:45.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:45.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:45.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:45.554+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:46.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:45.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:46.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:45.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:46.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:45.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:46.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:45.942+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:46.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:46.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:46.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:46.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:46.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:46.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:46.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:46.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:46.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:46.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:46.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:46.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:46.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:47.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:46.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:46.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:46.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:46.914+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:47.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:47.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:47.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:47.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:47.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:47.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:47.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:47.238+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:47.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:47.574+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:47.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:47.574+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:47.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:47.574+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:47.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:47.574+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:48.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:47.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:48.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:47.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:48.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:47.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:48.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:47.957+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:48.592 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:48.467+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24137 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:48.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:48.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:48.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:48.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:48.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:48.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:48.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:48.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:48.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:48.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:48.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:48.620+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:49.249 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:48.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:49.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:48.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:49.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:48.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:49.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:48.997+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:49.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:49.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:49.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:49.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:49.249+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:49.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:49.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:49.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:49.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:49.623+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:50.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:50.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:50.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:50.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:50.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:50.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:50.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:50.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:50.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:50.666+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:50.666+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:50.666+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:50.667+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:51.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:51.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:51.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:51.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:51.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:51.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:51.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:51.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:51.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:51.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:51.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:51.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:52.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:52.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:52.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:52.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:52.078+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:52.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:52.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:52.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:52.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:52.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:52.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:52.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:52.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:52.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:53.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:53.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:53.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:53.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:53.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:53.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:53.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:53.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:53.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:53.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:53.470+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24226 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:53.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:53.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:53.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:53.684+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:54.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:54.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:54.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:54.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:54.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:54.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:54.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:54.050+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:54.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:54.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:54.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:54.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:55.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:55.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:55.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:55.083+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:55.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:55.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:55.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:55.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:55.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:55.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:55.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:55.610+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:56.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:56.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:56.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:56.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:56.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:56.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:56.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:56.126+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:56.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:56.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:56.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:56.589+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:57.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:57.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:57.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:57.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:57.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:57.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:57.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:57.078+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:57.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:57.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:57.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:57.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:58.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:58.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:58.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:58.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:58.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:58.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:58.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:58.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:58.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:25:58.472+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24317 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:25:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:58.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:58.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:58.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:58.594+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:59.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:59.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:59.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:25:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:25:59.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:25:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:59.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:59.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:59.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:25:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:25:59.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:59.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:59.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:59.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:25:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:25:59.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:00.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:00.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:00.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:00.103+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:00.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:00.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:00.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:00.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:00.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:00.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:00.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:00.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:01.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:01.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:01.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:01.160+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:01.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:01.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:01.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:01.100+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:01.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:01.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:01.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:01.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:02.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:02.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:02.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:02.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:02.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:02.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:02.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:02.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:02.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:02.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:02.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:02.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:03.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:03.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:03.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:03.128+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:03.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:03.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:03.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:03.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:03.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:03.475+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24405 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:03.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:03.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:03.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:03.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:04.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:04.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:04.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:04.166+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:04.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:04.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:04.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:04.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:04.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:04.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:04.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:04.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:04.604+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:05.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:05.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:05.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:05.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:05.193+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:05.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:05.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:05.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:05.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:05.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:05.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:05.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:05.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:05.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:05.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:05.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:06.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:06.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:06.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:06.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:06.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:06.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:06.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:06.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:06.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:06.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:06.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:06.675+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:06.676+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:07.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:07.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:07.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:07.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:07.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:07.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:07.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:07.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:07.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:07.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:07.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:07.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:07.663+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:08.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:08.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:08.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:08.220+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:08.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:08.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:08.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:08.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:08.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:08.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:08.105+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:08.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:08.477+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24496 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:08.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:08.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:08.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:08.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:09.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:09.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:09.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:09.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:09.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:09.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:09.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:09.118+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:09.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:09.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:09.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:09.672+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:10.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:10.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:10.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:10.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:10.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:10.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:10.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:10.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:10.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:10.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:10.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:10.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:11.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:11.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:11.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:11.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:11.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:11.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:11.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:11.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:11.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:11.630+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:12.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:12.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:12.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:12.191+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:12.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:12.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:12.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:12.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:12.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:12.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:12.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:12.679+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:13.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:13.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:13.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:13.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:13.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:13.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:13.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:13.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:13.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:13.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:13.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:13.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:13.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:13.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:13.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:13.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:13.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:13.480+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24585 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:13.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:13.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:13.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:13.631+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:14.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:14.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:14.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:14.312+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:14.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:14.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:14.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:14.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:14.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:14.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:14.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:14.638+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:15.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:15.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:15.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:15.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:15.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:15.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:15.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:15.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:15.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:15.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:15.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:15.624+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:16.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:16.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:16.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:16.353+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:16.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:16.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:16.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:16.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:16.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:16.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:16.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:16.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:17.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:17.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:17.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:17.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:17.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:17.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:17.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:17.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:17.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:17.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:17.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:17.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:18.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:18.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:18.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:18.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:18.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:18.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:18.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:18.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:18.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:18.482+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24676 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:18.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:18.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:18.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:18.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:19.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:19.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:19.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:19.312+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:19.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:19.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:19.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:19.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:19.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:19.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:19.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:19.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:20.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:20.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:20.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:20.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:20.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:20.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:20.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:20.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:20.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:20.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:20.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:20.594+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:21.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:21.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:21.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:21.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:21.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:21.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:21.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:21.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:21.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:21.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:21.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:21.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:21.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:21.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:21.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:21.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:21.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:21.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:21.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:21.610+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:22.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:22.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:22.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:22.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:22.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:22.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:22.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:22.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:22.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:22.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:22.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:22.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:22.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:22.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:22.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:22.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:22.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:22.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:22.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:22.659+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:23.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:23.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:23.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:23.236+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:23.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:23.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:23.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:23.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:23.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:23.485+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24764 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:23.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:23.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:23.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:23.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:24.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:24.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:24.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:24.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:24.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:24.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:24.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:24.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:24.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:24.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:24.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:24.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:24.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:24.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:25.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:25.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:25.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:25.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:25.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:25.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:25.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:25.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:25.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:25.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:25.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:25.654+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:26.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:26.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:26.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:26.114+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:26.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:26.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:26.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:26.272+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:26.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:26.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:26.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:26.663+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:27.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:27.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:27.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:27.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:27.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:27.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:27.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:27.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:27.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:27.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:27.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:27.690+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:28.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:28.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:28.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:28.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:28.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:28.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:28.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:28.107+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:28.742 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:28.488+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24855 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:28.742 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:28.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:28.742 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:28.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:28.742 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:28.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:28.742 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:28.740+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:29.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:29.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:29.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:29.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:29.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:29.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:29.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:29.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:30.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:29.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:30.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:29.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:30.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:29.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:30.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:29.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:30.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:30.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:30.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:30.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:30.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:30.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:30.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:30.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:31.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:30.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:31.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:30.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:31.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:30.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:31.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:30.753+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:31.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:31.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:31.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:31.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:31.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:31.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:31.113+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:31.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:31.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:31.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:31.296+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:31.999 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:31.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:32.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:31.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:32.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:31.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:32.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:31.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:32.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:32.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:32.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:32.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:32.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:32.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:32.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:32.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:33.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:32.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:33.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:32.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:33.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:32.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:33.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:32.790+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:33.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:33.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:33.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:33.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:33.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:33.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:33.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:33.141+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:33.816 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:33.490+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24944 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:34.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:33.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:34.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:33.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:34.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:33.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:34.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:33.815+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:34.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:34.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:34.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:34.274+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:34.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:34.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:34.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:34.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:35.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:34.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:35.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:34.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:35.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:34.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:35.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:34.829+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:35.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:35.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:35.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:35.186+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:35.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:35.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:35.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:35.258+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:36.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:35.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:36.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:35.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:36.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:35.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:36.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:35.814+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:36.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:36.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:36.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:36.223+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:36.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:36.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:36.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:36.162+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:37.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:36.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:37.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:36.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:37.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:36.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:37.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:36.770+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:37.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:37.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:37.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:37.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:37.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:37.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:37.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:37.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:38.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:37.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:38.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:37.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:38.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:37.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:38.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:37.773+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:38.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:38.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:38.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:38.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:38.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:38.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:38.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:38.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:38.791 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:38.493+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25035 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:39.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:38.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:39.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:38.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:39.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:38.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:39.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:38.790+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:39.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:39.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:39.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:39.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:39.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:39.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:39.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:39.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:40.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:39.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:40.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:39.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:40.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:39.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:40.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:39.821+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:40.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:40.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:40.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:40.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:40.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:40.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:40.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:40.087+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:41.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:41.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:41.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:41.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:41.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:41.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:41.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:41.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:41.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:40.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:41.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:40.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:41.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:40.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:41.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:40.868+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:41.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:41.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:41.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:41.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:42.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:41.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:42.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:41.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:42.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:41.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:42.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:41.905+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:42.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:42.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:42.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:42.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:42.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:42.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:42.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:42.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:42.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:43.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:43.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:43.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:43.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:43.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:43.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:43.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:43.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:43.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:42.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:43.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:42.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:43.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:42.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:43.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:42.874+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:43.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:43.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:43.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:43.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:43.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:43.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:43.496+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25124 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:44.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:44.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:44.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:44.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:44.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:44.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:44.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:44.074+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:44.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:43.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:44.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:43.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:44.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:43.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:44.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:43.826+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:44.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:44.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:44.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:44.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:44.167+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:45.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:45.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:45.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:45.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:45.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:45.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:45.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:45.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:45.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:44.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:45.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:44.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:45.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:44.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:45.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:44.867+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:45.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:45.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:45.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:45.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:46.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:46.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:46.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:46.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:46.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:46.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:46.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:46.090+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:46.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:45.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:46.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:45.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:46.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:45.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:46.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:45.877+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:46.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:46.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:46.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:46.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:46.211+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:47.184 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:47.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:47.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:47.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:47.114+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:46.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:46.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:46.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:47.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:46.923+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:47.182+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:47.182+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:47.182+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:47.183+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:48.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:48.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:48.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:48.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:47.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:47.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:47.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:47.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:48.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:48.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:48.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:48.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:48.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:48.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:48.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:48.220+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:48.953 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:48.498+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25214 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:49.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:49.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:49.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:49.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:48.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:48.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:48.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:48.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:49.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:49.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:49.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:49.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:50.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:50.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:50.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:50.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:50.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:50.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:50.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:50.106+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:50.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:49.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:50.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:49.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:50.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:49.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:50.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:49.930+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:50.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:50.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:50.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:50.291+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:51.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:51.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:51.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:51.104+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:50.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:50.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:50.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:50.922+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:51.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:51.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:51.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:51.331+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:52.065 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:51.922+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:52.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:51.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:52.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:51.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:52.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:51.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:52.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:52.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:52.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:52.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:52.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:52.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:52.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:52.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:52.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:52.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:52.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:52.369+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:53.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:53.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:53.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:53.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:52.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:52.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:52.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:52.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:53.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:53.501+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25304 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:53.618 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:53.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:53.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:53.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:53.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:53.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:53.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:53.336+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:54.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:54.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:54.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:54.105+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:53.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:53.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:53.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:53.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:54.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:54.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:54.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:54.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:55.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:54.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:55.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:54.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:55.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:54.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:55.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:54.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:55.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:55.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:55.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:55.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:55.137+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:55.137+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:55.137+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:55.138+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:56.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:55.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:56.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:55.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:56.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:55.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:56.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:55.926+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:56.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:56.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:56.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:56.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:56.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:56.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:56.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:56.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:56.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:56.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:56.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:56.934+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:57.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:57.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:57.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:57.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:57.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:57.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:57.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:57.369+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:57.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:57.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:57.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:57.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:57.223+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:57.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:57.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:57.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:57.925+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:58.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:26:58.504+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25394 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:26:58.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:58.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:58.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:58.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:58.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:58.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:58.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:58.413+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:58.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:58.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:58.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:58.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:58.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:58.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:58.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:58.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:58.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:58.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:58.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:26:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:58.968+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:26:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:59.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:59.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:59.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:26:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:26:59.274+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:59.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:59.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:59.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:26:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:26:59.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:00.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:59.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:00.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:59.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:00.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:59.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:00.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:26:59.922+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:00.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:00.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:00.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:00.411+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:00.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:00.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:00.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:00.302+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:00.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:00.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:00.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:00.881+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:01.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:01.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:01.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:01.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:01.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:01.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:01.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:01.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:01.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:01.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:01.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:01.886+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:02.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:02.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:02.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:02.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:02.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:02.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:02.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:02.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:02.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:02.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:02.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:02.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:02.303+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:02.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:02.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:02.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:02.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:03.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:03.507+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25482 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:03.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:03.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:03.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:03.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:03.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:03.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:03.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:03.281+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:03.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:03.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:03.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:03.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:03.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:03.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:03.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:03.379+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:03.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:03.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:03.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:03.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:04.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:04.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:04.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:04.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:04.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:04.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:04.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:04.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:04.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:04.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:04.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:04.953+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:05.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:05.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:05.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:05.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:05.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:05.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:05.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:05.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:06.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:05.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:06.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:05.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:06.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:05.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:06.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:05.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:06.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:06.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:06.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:06.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:06.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:06.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:06.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:06.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:06.193+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:07.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:06.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:07.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:06.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:07.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:06.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:07.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:06.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:07.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:07.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:07.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:07.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:07.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:07.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:07.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:07.186+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:08.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:08.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:08.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:08.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:07.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:07.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:07.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:07.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:08.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:08.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:08.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:08.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:08.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:08.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:08.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:08.238+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:08.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:08.508+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25571 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:08.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:08.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:08.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:08.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:09.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:09.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:09.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:09.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:09.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:09.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:09.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:09.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:10.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:10.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:10.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:10.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:10.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:10.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:10.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:10.110+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:10.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:09.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:10.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:09.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:10.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:09.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:10.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:09.899+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:10.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:10.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:10.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:10.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:10.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:10.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:10.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:10.876+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:11.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:11.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:11.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:11.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:11.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:11.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:11.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:11.243+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:12.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:11.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:12.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:11.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:12.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:11.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:12.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:11.846+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:12.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:12.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:12.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:12.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:12.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:12.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:12.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:12.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:13.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:12.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:13.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:12.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:13.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:12.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:13.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:12.800+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:13.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:13.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:13.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:13.264+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:13.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:13.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:13.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:13.080+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:13.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:13.512+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25661 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:14.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:14.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:14.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:14.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:13.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:13.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:13.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:13.828+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:14.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:14.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:14.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:14.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:14.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:14.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:14.226+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:15.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:14.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:15.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:14.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:15.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:14.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:15.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:14.786+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:15.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:15.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:15.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:15.239+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:15.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:15.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:15.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:15.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:16.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:15.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:16.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:15.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:16.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:15.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:16.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:15.794+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:16.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:16.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:16.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:16.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:16.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:16.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:16.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:16.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:16.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:17.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:16.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:17.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:16.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:17.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:16.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:17.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:16.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:17.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:17.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:17.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:17.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:17.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:17.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:17.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:17.232+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:17.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:17.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:17.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:17.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:18.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:18.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:18.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:18.282+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:18.110+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:18.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:18.515+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25751 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:18.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:18.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:18.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:18.723+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:19.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:19.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:19.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:19.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:19.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:19.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:19.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:19.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:20.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:19.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:20.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:19.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:20.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:19.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:20.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:19.757+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:20.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:20.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:20.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:20.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:20.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:20.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:20.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:20.296+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:21.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:20.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:21.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:20.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:21.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:20.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:21.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:20.773+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:21.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:21.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:21.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:21.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:21.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:21.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:21.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:21.277+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:21.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:21.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:21.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:21.816+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:22.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:22.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:22.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:22.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:22.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:22.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:22.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:22.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:22.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:22.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:22.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:22.837+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:23.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:23.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:23.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:23.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:23.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:23.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:23.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:23.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:23.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:23.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:23.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:23.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:23.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:23.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:23.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:23.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:23.880 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:23.518+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25840 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:24.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:23.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:24.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:23.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:24.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:23.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:24.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:23.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:24.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:24.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:24.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:24.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:24.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:24.258+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:24.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:24.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:24.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:24.209+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:24.833+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:24.833+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:24.833+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:24.834+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:25.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:25.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:25.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:25.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:25.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:25.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:25.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:25.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:25.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:25.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:25.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:25.811+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:26.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:26.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:26.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:26.258+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:26.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:26.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:26.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:26.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:26.837+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:26.837+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:26.837+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:27.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:26.838+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:27.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:27.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:27.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:27.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:27.218+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:27.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:27.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:27.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:27.191+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:28.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:27.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:28.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:27.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:28.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:27.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:28.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:27.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:28.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:28.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:28.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:28.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:28.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:28.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:28.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:28.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:28.913 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:28.520+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25930 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:28.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:28.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:28.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:28.912+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:29.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:29.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:29.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:29.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:29.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:29.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:29.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:29.234+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:30.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:29.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:30.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:29.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:30.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:29.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:30.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:29.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:30.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:30.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:30.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:30.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:30.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:30.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:30.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:30.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:31.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:31.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:31.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:31.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:31.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:31.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:31.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:31.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:31.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:30.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:31.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:30.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:31.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:30.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:31.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:30.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:31.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:31.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:31.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:31.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:32.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:31.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:32.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:31.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:32.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:31.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:32.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:31.920+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:32.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:32.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:32.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:32.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:32.126+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:32.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:32.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:32.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:32.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:33.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:33.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:33.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:33.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:33.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:33.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:33.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:33.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:33.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:32.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:33.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:32.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:33.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:32.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:33.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:32.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:33.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:33.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:33.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:33.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:33.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:33.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:33.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:33.184+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:33.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:33.523+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26020 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:33.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:33.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:33.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:33.953+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:34.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:34.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:34.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:34.177+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:34.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:34.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:34.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:34.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:34.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:35.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:34.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:34.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:34.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:35.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:35.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:35.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:35.147+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:35.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:35.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:35.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:35.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:36.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:36.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:36.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:36.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:36.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:36.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:36.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:36.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:36.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:36.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:36.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:36.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:37.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:37.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:37.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:37.026+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:37.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:37.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:37.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:37.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:37.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:37.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:37.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:37.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:38.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:38.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:38.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:38.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:38.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:38.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:38.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:38.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:38.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:38.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:38.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:38.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:38.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:38.526+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26110 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:39.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:39.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:39.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:39.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:39.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:39.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:39.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:39.111+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:39.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:39.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:39.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:39.080+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:40.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:40.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:40.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:40.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:40.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:40.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:40.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:40.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:40.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:40.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:40.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:40.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:41.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:41.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:41.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:41.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:41.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:41.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:41.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:41.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:41.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:41.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:41.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:41.073+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:42.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:42.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:42.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:42.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:42.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:42.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:42.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:42.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:42.176+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:42.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:42.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:42.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:42.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:43.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:43.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:43.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:43.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:43.042+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:43.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:43.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:43.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:43.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:43.197+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:43.197+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:43.197+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:43.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:43.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:43.529+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26199 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:44.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:44.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:44.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:44.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:44.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:44.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:44.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:44.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:44.162+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:44.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:44.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:44.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:44.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:45.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:45.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:45.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:45.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:45.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:45.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:45.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:45.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:45.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:45.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:45.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:45.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:46.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:46.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:46.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:46.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:46.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:46.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:46.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:46.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:46.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:46.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:46.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:46.218+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:46.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:46.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:46.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:46.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:47.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:47.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:47.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:47.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:47.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:47.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:47.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:47.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:47.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:47.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:47.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:47.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:48.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:48.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:48.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:48.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:48.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:48.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:48.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:48.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:48.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:48.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:48.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:48.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:48.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:48.532+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26289 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:49.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:49.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:49.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:49.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:49.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:49.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:49.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:49.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:49.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:49.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:49.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:49.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:49.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:50.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:50.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:50.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:50.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:50.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:50.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:50.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:50.175+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:50.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:50.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:50.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:50.140+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:50.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:50.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:50.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:50.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:50.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:51.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:51.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:51.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:51.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:51.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:51.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:51.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:51.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:51.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:51.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:51.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:51.146+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:52.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:52.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:52.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:52.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:52.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:52.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:52.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:52.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:52.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:52.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:52.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:52.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:53.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:53.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:53.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:53.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:53.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:53.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:53.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:53.091+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:53.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:53.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:53.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:53.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:53.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:53.535+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26379 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:54.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:54.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:54.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:54.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:54.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:54.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:54.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:54.109+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:54.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:54.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:54.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:54.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:55.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:55.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:55.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:55.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:55.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:55.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:55.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:55.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:55.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:55.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:55.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:55.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:56.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:56.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:56.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:56.150+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:56.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:56.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:56.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:56.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:56.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:56.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:56.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:56.090+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:57.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:57.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:57.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:57.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:57.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:57.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:57.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:57.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:57.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:57.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:57.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:57.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:58.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:58.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:58.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:58.162+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:58.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:58.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:58.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:58.099+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:58.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:58.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:58.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:58.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:58.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:27:58.537+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26468 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:27:59.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:59.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:59.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:59.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:59.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:59.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:27:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:27:59.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:27:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:59.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:59.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:59.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:27:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:27:59.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:59.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:59.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:59.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:27:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:27:59.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:00.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:00.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:00.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:00.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:00.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:00.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:00.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:00.032+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:00.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:00.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:00.146+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:00.147+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:01.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:01.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:01.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:01.140+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:01.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:01.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:01.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:01.033+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:02.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:01.987+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:02.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:02.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:02.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:02.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:02.176+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:02.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:02.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:02.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:02.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:02.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:03.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:03.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:03.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:03.191+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:03.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:03.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:03.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:03.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:02.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:02.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:02.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:02.995+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:03.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:03.540+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26557 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:04.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:04.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:04.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:04.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:04.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:04.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:04.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:04.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:04.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:03.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:04.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:03.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:04.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:03.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:04.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:03.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:04.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:04.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:04.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:04.234+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:04.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:04.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:04.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:04.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:05.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:05.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:05.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:05.128+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:05.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:05.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:05.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:05.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:05.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:05.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:05.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:05.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:06.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:06.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:06.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:06.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:06.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:06.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:06.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:06.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:06.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:06.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:06.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:06.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:06.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:06.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:06.938+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:07.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:07.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:07.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:07.140+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:07.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:07.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:07.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:07.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:08.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:08.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:08.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:08.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:08.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:08.191+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:07.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:07.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:07.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:07.984+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:08.556 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:08.543+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26642 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:08.556 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:08.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:08.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:08.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:08.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:08.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:08.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:08.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:09.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:09.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:09.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:09.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:09.212+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:08.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:08.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:08.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:09.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:08.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:09.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:09.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:09.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:09.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:10.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:10.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:10.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:10.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:10.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:10.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:10.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:10.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:10.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:09.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:10.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:09.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:10.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:09.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:10.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:09.959+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:10.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:10.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:10.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:10.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:11.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:10.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:11.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:10.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:11.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:10.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:11.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:10.922+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:11.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:11.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:11.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:11.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:11.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:11.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:11.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:11.192+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:12.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:11.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:12.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:11.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:12.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:11.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:12.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:11.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:12.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:12.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:12.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:12.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:12.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:12.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:12.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:12.168+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:12.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:12.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:12.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:12.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:12.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:13.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:13.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:13.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:13.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:13.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:13.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:13.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:13.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:13.672 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:13.546+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26731 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:13.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:13.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:13.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:13.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:14.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:14.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:14.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:14.127+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:14.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:14.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:14.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:14.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:14.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:14.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:14.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:14.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:15.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:14.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:15.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:14.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:15.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:14.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:15.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:14.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:15.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:15.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:15.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:15.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:15.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:15.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:15.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:15.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:16.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:15.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:16.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:15.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:16.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:15.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:16.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:15.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:16.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:16.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:16.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:16.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:16.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:16.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:16.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:16.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:16.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:16.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:16.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:16.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:16.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:16.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:16.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:16.238+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:17.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:16.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:17.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:16.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:17.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:16.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:17.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:16.876+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:17.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:17.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:17.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:17.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:17.199+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:17.199+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:17.199+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:17.200+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:18.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:17.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:18.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:17.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:18.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:17.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:18.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:17.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:18.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:18.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:18.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:18.158+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:18.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:18.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:18.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:18.160+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:18.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:18.549+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26821 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:19.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:19.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:19.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:19.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:19.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:19.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:19.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:19.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:18.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:18.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:18.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:18.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:20.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:20.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:20.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:20.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:20.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:20.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:20.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:20.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:20.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:20.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:20.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:20.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:20.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:20.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:20.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:20.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:20.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:19.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:20.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:19.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:20.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:19.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:20.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:19.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:21.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:21.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:21.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:21.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:21.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:21.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:21.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:21.079+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:21.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:20.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:21.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:20.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:21.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:20.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:21.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:20.893+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:21.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:21.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:21.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:21.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:22.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:21.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:22.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:21.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:22.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:21.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:22.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:21.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:22.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:22.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:22.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:22.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:22.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:22.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:22.163+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:22.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:22.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:22.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:22.036+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:23.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:23.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:23.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:23.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:23.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:23.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:23.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:23.053+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:22.913+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:22.913+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:22.913+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:22.914+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:23.884 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:23.552+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26910 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:23.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:23.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:23.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:23.883+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:24.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:24.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:24.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:24.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:24.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:24.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:24.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:24.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:24.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:25.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:25.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:25.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:25.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:25.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:25.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:25.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:25.040+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:24.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:24.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:24.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:24.909+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:25.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:25.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:25.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:25.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:26.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:26.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:26.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:26.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:26.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:26.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:26.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:26.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:26.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:26.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:26.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:26.969+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:27.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:27.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:27.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:27.154+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:27.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:27.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:27.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:27.053+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:28.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:28.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:28.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:28.034+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:28.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:28.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:28.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:28.175+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:28.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:28.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:28.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:28.010+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:28.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:28.555+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27000 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:29.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:29.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:29.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:29.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:29.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:29.039+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:29.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:29.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:29.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:29.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:29.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:29.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:29.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:29.006+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:30.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:30.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:30.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:30.035+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:30.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:30.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:30.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:30.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:30.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:30.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:30.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:30.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:31.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:31.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:31.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:31.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:31.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:31.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:31.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:31.198+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:31.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:31.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:31.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:31.049+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:32.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:32.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:32.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:32.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:32.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:32.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:32.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:32.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:32.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:32.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:32.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:32.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:33.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:33.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:33.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:33.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:33.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:33.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:33.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:33.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:33.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:33.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:33.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:33.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:33.557+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27089 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:34.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:34.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:34.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:34.131+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:34.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:34.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:34.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:34.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:34.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:34.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:34.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:34.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:35.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:35.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:35.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:35.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:35.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:35.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:35.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:35.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:35.026+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:35.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:35.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:35.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:35.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:36.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:36.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:36.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:36.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:36.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:36.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:36.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:36.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:36.193+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:36.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:36.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:36.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:36.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:37.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:37.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:37.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:37.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:37.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:37.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:37.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:37.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:37.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:37.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:37.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:37.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:38.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:38.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:38.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:38.153+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:38.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:38.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:38.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:38.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:38.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:38.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:38.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:38.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:38.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:38.560+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27179 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:39.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:39.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:39.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:39.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:39.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:39.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:39.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:39.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:39.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:39.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:39.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:39.025+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:39.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:39.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:39.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:39.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:40.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:40.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:40.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:40.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:40.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:40.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:40.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:40.109+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:40.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:40.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:40.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:40.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:41.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:41.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:41.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:41.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:41.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:41.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:41.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:41.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:41.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:41.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:41.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:41.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:41.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:41.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:41.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:41.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:41.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:41.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:41.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:41.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:41.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:42.336 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:42.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:42.336 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:42.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:42.336 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:42.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:42.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:42.115+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:42.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:42.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:42.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:42.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:42.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:42.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:42.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:42.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:43.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:43.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:43.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:43.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:43.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:43.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:43.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.991+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:43.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:42.992+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:43.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:43.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:43.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:43.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:43.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:43.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:43.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:43.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:43.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:43.563+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27268 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:44.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:44.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:44.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:44.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:44.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:44.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:44.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:44.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:43.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:43.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:43.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:43.991+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:44.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:44.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:44.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:44.285+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:45.315 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:44.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:45.316 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:44.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:45.316 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:44.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:45.316 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:44.995+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:45.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:45.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:45.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:45.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:45.317 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:45.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:45.317 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:45.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:45.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:45.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:45.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:45.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:46.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:46.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:46.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:46.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:46.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:46.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:46.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:46.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:46.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:45.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:46.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:45.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:46.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:45.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:46.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:45.990+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:46.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:46.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:46.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:46.361+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:46.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:46.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:46.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:46.943+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:47.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:47.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:47.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:47.063+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:47.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:47.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:47.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:47.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:47.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:47.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:47.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:47.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:48.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:48.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:48.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:48.024+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:48.695 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:48.566+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27357 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:48.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:48.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:48.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:48.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:48.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:48.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:48.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:48.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:48.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:48.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:48.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:48.889+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:49.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:49.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:49.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:49.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:49.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:49.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:49.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:49.046+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:49.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:49.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:49.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:49.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:50.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:50.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:50.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:50.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:50.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:50.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:50.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:50.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:50.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:49.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:50.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:49.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:50.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:49.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:50.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:49.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:50.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:50.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:50.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:50.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:50.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.018+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:50.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:50.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:50.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:50.798+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:51.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:51.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:51.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:51.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:51.980+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:51.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:51.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:51.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:51.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:52.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:52.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:52.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:52.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:52.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:52.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:52.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:52.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:52.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:52.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:52.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:52.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:53.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:53.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:53.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:53.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:53.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:53.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:53.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:53.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:53.875 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:53.570+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27446 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:54.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:54.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:54.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:54.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:53.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:53.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:53.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:53.873+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:54.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:54.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:54.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:54.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:55.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:55.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:55.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:55.034+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:54.901+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:54.901+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:54.901+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:54.901+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:55.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:55.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:55.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:55.334+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:56.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:55.855+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:55.855+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:55.855+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:55.855+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:56.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:56.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:56.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:56.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:56.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:56.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:56.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:56.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:56.844+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:57.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:57.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:57.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:57.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:57.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:57.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:57.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:57.949+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:57.826+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:57.826+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:57.826+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:57.827+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:58.700 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:28:58.573+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27535 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:28:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:58.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:58.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:58.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:58.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:58.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:58.385+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:58.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:58.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:58.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:58.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:58.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:58.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:28:58.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:58.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:28:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:58.955+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:58.955+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:58.955+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:28:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:28:58.956+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:59.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:59.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:59.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:28:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:28:59.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:00.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:00.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:00.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:00.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.005+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:00.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:59.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:00.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:59.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:00.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:59.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:00.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:28:59.837+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:00.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:00.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:00.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:00.333+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:00.985+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:00.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:00.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:00.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:00.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:01.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:01.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:01.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:01.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:01.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:01.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:01.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:01.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:01.979+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:01.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:01.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:01.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:01.847+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:02.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:02.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:02.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:02.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:02.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:02.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:02.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:02.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:02.949+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:02.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:02.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:02.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:02.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:03.627 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:03.576+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27624 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:03.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:03.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:03.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:03.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:03.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:03.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:03.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:03.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:03.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:03.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:03.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:03.853+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:04.329 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:03.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:04.329 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:03.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:04.329 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:03.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:04.329 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:03.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:04.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:04.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:04.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:04.328+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:04.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:04.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:04.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:04.876+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:05.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:05.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:05.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:05.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:05.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:05.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:05.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:05.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:05.600 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:05.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:05.600 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:05.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:06.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:06.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:06.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:06.025+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:05.888+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:05.888+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:05.888+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:05.889+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:06.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:06.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:06.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:06.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:06.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:07.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:07.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:07.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:07.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:06.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:06.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:06.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:06.922+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:07.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:07.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:07.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:07.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:08.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:08.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:08.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:08.050+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:07.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:07.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:07.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:07.905+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:08.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:08.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:08.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:08.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:08.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:08.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:08.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:08.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:08.881 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:08.579+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27713 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:09.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:09.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:09.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:09.036+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:08.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:08.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:08.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:08.880+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:09.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:09.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:09.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:09.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:10.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:10.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:10.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:10.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:10.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:10.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:10.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:10.007+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:10.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:09.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:10.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:09.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:10.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:09.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:10.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:09.923+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:10.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:10.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:10.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:10.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:11.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:10.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:10.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:10.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:10.898+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:11.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:11.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:11.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:11.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:11.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:11.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:11.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:11.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:11.872+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:12.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:12.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:11.982+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:12.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:12.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:12.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:12.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:12.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:12.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:12.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:12.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:12.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:12.901+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:12.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:12.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:12.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:12.990+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:13.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:13.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:13.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:13.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:13.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:13.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:13.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:13.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:13.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:13.582+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27803 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:13.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:13.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:13.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:13.944+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:13.862+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:13.862+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:13.862+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:13.863+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:14.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:14.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:14.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:14.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:14.310+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:14.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:14.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:15.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:14.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:15.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:14.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:14.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:14.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:14.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:14.855+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:15.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:15.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:15.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:15.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:15.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:16.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:15.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:15.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:15.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:15.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:15.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:15.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:15.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:15.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:16.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:16.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:16.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:16.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:16.258+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:16.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:16.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:16.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:16.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:16.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:16.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:16.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:17.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:16.776+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:17.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:17.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:17.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:17.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:17.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:18.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:17.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:18.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:17.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:18.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:17.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:18.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:17.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:18.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:17.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:18.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:17.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:18.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:17.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:18.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:17.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:18.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:18.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:18.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:18.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:18.943 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:18.585+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27892 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:18.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:18.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:18.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:18.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:18.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:18.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:18.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:18.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:18.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:18.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:18.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:18.942+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:19.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:19.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:19.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:19.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:20.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:19.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:20.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:19.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:20.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:19.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:20.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:19.931+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:20.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:19.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:20.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:19.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:20.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:19.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:20.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:19.763+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:20.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:20.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:20.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:20.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:20.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:20.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:20.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:20.801+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:20.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:20.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:20.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:20.930+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:21.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:21.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:21.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:21.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:21.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:21.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:21.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:21.844+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:21.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:21.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:21.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:21.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:22.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:22.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:22.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:22.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:22.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:22.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:22.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:22.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:22.829+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:22.829+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:22.829+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:22.830+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:23.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:23.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:23.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:23.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:23.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:23.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:23.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:23.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:23.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:23.588+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27980 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:23.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:23.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:23.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:23.935+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:23.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:23.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:23.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:24.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:23.872+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:24.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:24.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:24.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:24.322+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:25.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:24.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:24.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:24.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:24.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:24.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:24.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:24.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:24.899+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:25.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:25.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:25.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:25.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:25.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:25.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:25.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:25.888+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:25.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:25.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:25.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:25.915+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:26.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:26.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:26.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:26.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:26.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:26.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:26.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:26.945+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:26.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:26.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:26.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:26.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:27.268+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:27.268+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:27.268+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:27.269+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:27.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:27.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:27.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:27.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:27.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:27.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:28.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:27.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:28.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:27.912+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:28.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:28.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:28.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:28.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:28.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:28.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:28.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:28.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:28.919 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:28.591+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28070 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:28.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:28.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:28.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:28.918+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:28.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:28.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:28.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:28.962+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:29.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:29.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:29.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:29.231+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:30.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:29.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:30.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:29.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:30.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:29.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:30.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:29.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:30.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:29.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:30.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:29.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:30.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:29.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:30.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:29.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:30.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:30.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:30.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:30.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:30.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:30.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:30.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:30.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:30.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:30.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:30.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:30.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:31.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:31.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:31.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:31.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:32.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:31.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:32.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:31.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:32.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:31.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:32.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:31.856+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:32.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:32.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:32.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:32.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:32.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:32.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:32.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:32.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:32.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:32.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:32.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:32.862+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:32.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:33.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:33.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:33.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:33.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:33.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:33.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:33.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:33.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:33.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:33.594+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28159 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:33.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:33.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:33.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:33.958+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:33.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:33.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:33.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:33.857+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:34.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:34.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:34.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:34.363+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:34.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:34.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:34.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:34.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:34.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:34.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:34.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:34.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:35.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:35.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:35.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:35.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:36.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:35.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:35.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:35.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:35.908+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:35.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:35.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:36.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:35.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:36.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:35.957+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:36.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:36.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:36.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:36.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:36.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:36.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:36.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:36.952+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:36.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:36.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:36.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:36.941+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:37.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:37.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:37.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:37.326+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:37.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:37.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:37.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:37.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:37.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:37.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:37.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:37.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:38.624 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:38.597+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28248 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:38.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:38.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:38.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:38.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:38.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:38.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:38.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:38.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:39.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:38.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:38.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:39.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:38.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:39.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:38.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:39.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:39.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:39.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:39.333+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:40.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:40.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:40.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:40.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:39.999+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:40.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:39.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:40.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:39.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:40.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:39.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:40.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:39.885+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:40.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:40.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:40.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:40.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:41.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:40.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:40.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:40.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:40.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:41.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:41.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:41.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:41.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:42.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:41.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:42.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:41.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:42.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:41.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:42.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:41.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:42.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:42.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:42.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:42.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:41.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:42.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:42.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:42.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:42.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:42.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:42.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:42.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:42.956+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:42.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:42.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:42.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:42.867+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:43.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:43.599+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28336 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:43.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:43.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:43.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:43.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:43.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:43.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:43.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:43.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:43.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:43.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:43.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:43.973+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:44.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:44.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:44.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:44.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:44.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:44.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:44.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:44.886+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:44.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:44.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:44.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:44.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:45.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:45.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:45.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:45.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:46.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:45.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:45.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:45.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:45.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:45.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:45.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:45.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:45.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:46.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:46.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:46.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:46.401+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:46.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:46.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:46.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:46.889+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:47.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:47.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:47.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:47.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:47.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:47.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:47.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:47.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:48.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:48.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:48.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:48.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:47.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:47.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:47.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:47.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:48.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:48.603+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28426 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:48.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:48.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:48.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:48.463+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:48.989 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:48.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:48.989 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:48.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:48.989 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:48.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:48.989 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:48.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:49.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:49.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:49.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:49.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:49.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:49.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:49.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:49.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:49.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:49.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:49.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:49.418+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:50.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:50.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:50.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:50.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:50.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:49.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:49.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:49.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:50.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:49.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:50.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:50.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:50.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:50.465+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:51.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:51.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:51.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:51.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:51.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:51.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:51.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:51.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:51.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:51.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:51.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:51.494+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:52.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:52.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:52.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:52.143+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:52.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:52.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:52.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:52.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:52.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:52.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:52.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:52.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:52.480+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:53.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:53.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:53.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:53.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:53.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:53.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:53.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:53.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:53.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:53.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:53.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:53.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.012+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:53.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:53.606+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28515 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:53.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:53.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:53.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:53.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:54.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:54.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:54.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:54.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:53.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:54.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:54.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:54.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:54.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:54.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:54.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:54.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:54.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:54.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:54.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:54.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:54.496+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:55.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:54.935+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:55.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:54.935+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:55.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:54.935+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:55.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:54.936+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:55.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:55.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:55.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:55.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:55.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:55.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:55.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:55.543+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:56.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:56.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:56.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:56.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:55.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:55.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:55.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:55.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:56.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:56.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:56.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:56.564+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:57.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:56.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:57.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:56.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:57.262 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:56.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:57.263 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:56.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:57.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:57.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:57.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:57.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:57.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:57.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:57.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:57.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:57.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:57.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:57.958 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:57.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:58.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:57.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:58.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:57.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:58.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:57.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:58.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:57.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:58.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:58.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:58.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:58.218+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:58.933 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:29:58.610+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28606 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:29:58.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:58.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:58.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:58.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:58.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:58.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:58.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:58.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:59.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:58.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:59.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:58.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:59.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:58.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:29:59.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:58.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:29:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:59.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:59.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:59.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:29:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:29:59.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:59.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:59.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:59.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:59.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:59.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:59.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:29:59.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:29:59.606+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:00.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:59.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:00.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:59.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:00.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:59.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:00.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:29:59.906+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:00.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:00.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:00.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:00.203+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:00.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:00.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:00.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:00.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:00.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:00.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:00.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:00.578+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:01.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:00.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:01.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:00.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:01.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:00.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:01.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:00.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:01.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:01.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:01.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:01.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:01.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:01.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:01.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:01.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:01.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:01.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:01.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:01.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:01.590+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:02.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:01.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:02.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:01.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:02.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:01.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:02.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:01.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:02.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:02.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:02.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:02.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:02.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:02.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:02.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:02.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:02.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:02.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:02.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:02.543+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:03.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:02.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:03.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:02.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:03.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:02.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:03.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:02.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:03.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:03.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:03.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:03.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:03.952 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:03.613+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28694 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:03.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:03.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:03.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:03.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:03.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:03.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:03.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:03.539+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:03.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:03.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:03.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:03.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:04.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:04.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:04.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:04.154+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:04.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:04.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:04.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:04.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:04.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:04.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:04.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:04.568+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:05.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:05.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:05.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:05.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:05.128+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:04.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:04.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:04.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:04.910+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:05.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:05.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:05.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:05.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:05.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:05.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:05.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:05.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:05.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:05.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:05.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:05.899+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:06.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:06.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:06.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:06.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:06.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:06.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:06.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:06.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:06.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:06.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:06.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:06.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:06.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:07.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:07.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:07.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:07.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:06.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:06.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:06.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:06.938+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:07.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:07.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:07.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:07.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:07.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:07.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:07.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:07.577+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:08.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:07.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:07.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:07.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:07.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:08.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:08.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:08.140+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:08.141+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:08.951 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:08.616+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28783 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:08.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:08.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:08.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:08.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:08.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:08.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:08.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:08.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:09.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:09.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:09.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:09.169+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:08.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:08.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:08.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:08.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:09.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:09.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:09.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:09.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:09.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:09.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:09.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:09.567+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:10.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:09.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:10.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:09.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:10.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:09.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:10.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:09.903+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:10.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:10.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:10.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:10.215+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:10.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:10.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:10.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:10.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:10.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:10.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:10.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:10.597+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:11.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:10.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:11.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:10.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:11.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:10.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:11.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:10.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:11.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:11.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:11.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:11.201+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:11.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:11.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:11.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:11.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:11.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:11.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:11.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:11.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:12.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:11.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:12.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:11.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:12.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:11.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:12.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:11.939+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:12.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:12.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:12.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:12.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:12.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:12.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:12.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:12.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:12.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:12.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:12.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:12.557+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:13.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:12.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:13.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:12.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:13.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:12.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:13.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:12.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:13.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:13.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:13.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:13.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:13.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:13.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:13.920 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:13.619+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28871 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:13.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:13.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:13.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:13.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:13.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:13.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:13.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:13.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:14.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:13.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:13.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:13.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:13.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:14.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:14.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:14.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:14.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:14.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:14.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:14.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:14.249+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:14.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:14.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:14.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:14.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:14.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:14.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:14.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:14.502+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:14.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:14.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:14.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:14.909+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:15.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:15.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:15.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:15.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:15.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:15.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:15.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:15.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:15.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:15.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:15.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:15.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:15.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:15.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:15.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:15.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:16.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:15.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:15.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:15.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:15.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:16.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:16.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:16.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:16.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:16.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:16.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:16.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:16.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:16.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:16.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:16.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:16.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:16.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:16.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:16.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:16.531+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:16.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:16.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:16.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:16.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:17.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:17.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:17.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:17.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:17.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:17.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:17.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:17.264+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:17.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:17.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:17.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:17.559+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:18.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:18.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:18.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:18.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:17.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:17.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:17.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:17.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:18.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:18.622+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28961 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:18.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:18.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:18.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:18.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:19.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:18.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:18.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:18.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:18.970+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:19.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:19.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:19.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:19.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:19.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:19.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:19.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:19.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:19.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:19.270+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:20.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:19.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:20.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:19.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:20.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:19.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:20.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:19.925+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:20.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:20.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:20.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:20.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:20.270+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:20.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:20.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:20.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:20.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:20.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:20.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:20.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:20.487+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:21.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:20.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:20.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:20.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:20.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:21.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:21.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:21.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:21.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:21.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:21.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:21.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:21.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:21.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:22.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:21.883+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:22.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:21.883+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:22.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:21.883+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:22.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:21.884+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:22.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:22.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:22.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:22.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:22.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:22.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:22.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:22.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:22.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:22.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:22.522+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:22.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:22.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:22.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:22.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:23.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:23.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:23.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:23.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:23.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:23.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:23.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:23.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:23.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:23.625+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29051 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:23.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:23.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:23.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:23.880 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:23.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:23.881 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:23.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:23.881 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:23.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:23.881 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:23.540+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:24.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:23.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:24.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:24.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:24.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:24.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:24.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:24.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:24.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:24.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:24.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:24.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:24.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:24.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:24.589+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:25.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:24.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:25.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:24.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:25.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:24.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:24.854+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:25.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:25.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:25.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:25.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:25.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:25.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:25.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:25.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:25.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:25.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:25.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:25.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:25.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:25.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:25.611+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:26.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:25.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:25.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:25.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:25.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:26.567 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:26.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:26.567 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:26.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:26.567 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:26.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:26.567 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:26.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:26.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:26.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:26.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:26.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:26.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:26.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:26.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:26.565+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:26.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:26.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:26.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:26.850+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:27.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:27.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:27.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:27.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:27.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:27.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:27.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:27.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:27.897 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:27.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:27.897 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:27.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:27.897 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:27.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:27.897 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:27.554+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:28.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:27.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:28.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:27.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:28.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:27.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:28.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:27.895+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:28.219+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:28.219+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:28.219+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:28.219+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:28.914 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:28.629+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29140 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:28.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:28.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:28.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:28.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:28.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:28.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:28.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:28.519+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:29.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:28.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:29.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:28.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:29.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:28.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:29.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:28.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:29.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:29.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:29.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:29.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:29.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:29.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:29.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:29.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:29.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:29.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:29.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:29.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:30.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:29.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:30.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:29.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:30.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:29.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:30.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:29.909+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:30.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:30.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:30.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:30.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:30.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:30.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:30.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:30.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:30.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:30.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:30.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:30.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:30.562+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:30.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:30.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:30.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:31.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:30.894+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:31.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:31.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:31.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:31.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:31.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:31.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:31.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:31.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:31.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:31.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:31.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:31.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:31.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:31.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:31.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:31.560+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:31.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:31.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:31.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:31.849+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:32.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:32.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:32.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:32.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:32.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:32.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:32.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:32.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:32.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:32.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:32.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:32.601+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:32.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:32.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:32.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:32.814+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:33.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:33.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:33.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:33.214+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:33.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:33.632+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29230 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:33.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:33.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:33.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:33.562+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:33.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:33.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:33.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:33.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:34.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:34.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:34.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:34.192+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:34.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:34.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:34.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:34.555+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:34.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:34.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:34.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:34.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:35.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:35.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:35.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:35.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:35.834 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:35.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:35.834 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:35.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:35.834 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:35.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:35.834 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:35.578+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:36.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:35.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:36.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:35.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:36.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:35.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:36.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:35.832+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:36.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:36.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:36.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:36.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:36.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:36.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:36.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:36.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:36.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:36.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:36.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:36.546+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:37.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:36.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:37.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:36.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:37.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:36.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:37.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:36.825+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:37.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:37.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:37.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:37.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:37.187+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:37.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:37.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:37.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:37.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:37.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:37.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:37.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:37.520+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:38.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:37.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:38.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:37.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:38.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:37.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:38.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:37.874+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:38.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:38.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:38.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:38.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:38.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:38.830 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:38.636+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29321 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:38.830 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:38.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:38.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:38.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:38.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:38.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:38.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:38.546+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:38.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:38.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:38.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:38.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:38.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:38.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:38.829+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:39.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:39.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:39.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:39.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:39.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:39.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:39.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:39.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:39.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:39.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:39.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:39.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:39.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:39.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:39.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:39.544+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:40.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:39.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:40.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:39.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:40.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:39.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:40.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:39.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:40.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:40.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:40.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:40.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:40.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:40.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:40.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:40.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:40.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:40.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:40.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:40.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:40.510+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:40.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:40.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:40.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:40.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:41.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:41.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:41.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:41.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:41.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:41.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:41.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:41.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:41.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:41.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:41.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:41.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:41.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:41.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:41.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:41.514+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:42.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:41.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:42.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:41.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:42.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:41.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:42.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:41.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:42.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:42.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:42.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:42.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:42.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:42.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:42.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:42.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:42.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:42.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:42.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:42.556+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:43.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:42.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:43.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:42.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:43.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:42.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:43.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:42.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:43.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:43.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:43.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:43.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:43.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:43.904 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:43.639+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29410 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:43.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:43.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:43.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:43.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:43.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:43.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:43.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:43.551+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:43.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:43.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:43.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:43.942+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:44.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:44.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:44.139+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:44.140+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:44.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:44.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:44.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:44.570+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:45.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:45.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:45.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:45.132+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:44.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:44.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:44.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:44.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:45.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:45.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:45.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:45.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:45.591+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:46.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:46.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:46.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:46.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:46.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:46.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:46.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:46.018+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:46.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:46.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:46.563+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:46.564+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:47.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:47.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:47.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:47.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:47.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:47.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:47.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:47.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:47.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:47.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:47.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:47.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:47.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:48.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:48.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:48.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:48.048+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:48.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:48.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:48.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:48.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:48.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:48.642+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29500 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:48.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:48.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:48.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:48.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:49.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:49.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:49.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:49.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.021+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:49.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:49.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:49.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:49.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:49.577+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:49.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:50.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:50.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:50.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:50.578+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:51.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:50.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:50.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:50.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:50.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:50.986+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:51.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:51.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:51.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:51.615+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:51.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:51.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:51.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:51.973+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:51.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:51.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:51.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:51.991+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:52.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:52.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:52.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:52.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:52.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:52.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:52.937 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:52.615+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:52.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:52.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:52.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:52.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:52.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:52.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:52.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:52.958+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:53.905 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:53.645+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29590 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:53.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:53.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:53.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:53.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:53.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:53.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:53.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:53.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:53.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:53.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:53.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:53.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:53.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:53.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:53.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:53.922+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:54.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:54.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:54.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:54.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:54.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:54.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:54.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:54.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:54.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:54.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:54.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:54.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:54.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:54.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:54.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:54.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:54.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:54.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:54.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:54.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:55.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:55.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:55.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:55.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:55.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:55.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:55.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:55.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:56.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:55.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:56.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:55.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:56.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:55.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:56.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:55.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:56.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:55.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:56.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:55.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:56.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:55.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:56.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:55.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:56.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:56.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:56.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:56.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:56.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:56.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:56.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:56.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:56.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:56.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:56.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:56.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:56.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:56.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:56.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:56.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:57.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:57.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:57.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:57.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:57.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:57.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:27.095302+0000 (oldest deadline 2024-02-20T08:20:47.095302+0000) 2024-02-20T08:30:57.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:57.685+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:57.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:57.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:57.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:57.969+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:57.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:57.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:57.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:57.958+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:58.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:58.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:58.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:30:58.649+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29680 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:30:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:30:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:58.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:58.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:58.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:58.913+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:30:59.654+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:30:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:59.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:59.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:59.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:25.440555+0000 (oldest deadline 2024-02-20T08:20:45.440555+0000) 2024-02-20T08:30:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:30:59.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:30:59.995+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:00.641+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:00.853+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:01.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:01.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:01.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:01.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:01.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:01.671+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:01.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:02.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:02.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:02.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:02.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:02.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:02.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:02.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:02.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:02.678+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:02.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:03.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:03.653+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29769 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:03.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:03.871+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:04.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:20:34.693188+0000 (oldest deadline 2024-02-20T08:20:54.693188+0000) 2024-02-20T08:31:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:03.998+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:04.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:04.841+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:05.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:05.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:05.594+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:06.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:05.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:06.574+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:06.797+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:07.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:07.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:07.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:07.531+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:08.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:07.845+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:08.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:08.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:08.656+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29859 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:08.889 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:08.545+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:09.135+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:08.886+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:09.882 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:09.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:10.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:09.879+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:10.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:10.119+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:10.897 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:10.499+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:11.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:10.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:11.115+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:11.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:11.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:12.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:11.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:12.891 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:12.530+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:13.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:13.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:12.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:13.785 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:13.659+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29949 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:13.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:13.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:14.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:13.939+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:14.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:14.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:14.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:14.551+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:14.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:15.137+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:15.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:15.576+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:15.931+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:16.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:16.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:16.540+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:17.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:16.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:17.521+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:17.988+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:18.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:18.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:18.663+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30039 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:18.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:18.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:18.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:18.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:18.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:18.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:18.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:18.479+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:19.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:18.968+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:19.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:19.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:19.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:19.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:19.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:19.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:20.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:19.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:20.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:20.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:20.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:20.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:20.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:20.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:20.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:20.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:20.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:20.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:21.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:20.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:21.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:20.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:21.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:20.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:21.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:20.920+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:21.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:21.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:21.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:21.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:21.747 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:21.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:21.747 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:21.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:21.747 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:21.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:22.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:21.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:22.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:21.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:22.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:21.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:22.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:21.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:22.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:22.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:22.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:22.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:22.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:22.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:22.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:22.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:22.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:23.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:22.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:23.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:22.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:23.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:22.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:23.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:22.915+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:23.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:23.191+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:23.791 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:23.665+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30128 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:23.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:23.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:23.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:23.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:23.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:24.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:23.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:24.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:23.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:24.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:23.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:24.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:23.920+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:24.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:24.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:24.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:24.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:24.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:24.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:24.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:24.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:24.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:24.501+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:25.126+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:24.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:24.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:24.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:24.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:25.886 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:25.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:25.886 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:25.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:25.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:25.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:25.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:25.491+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:26.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:25.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:26.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:25.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:26.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:25.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:26.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:25.884+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:26.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:26.171+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:26.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:26.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:26.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:26.467+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:27.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:26.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:27.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:26.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:27.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:26.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:27.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:26.912+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:27.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:27.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:27.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:27.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:27.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:27.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:27.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:27.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:27.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:27.512+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:28.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:27.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:28.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:27.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:28.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:27.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:28.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:27.882+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:28.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:28.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:28.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:28.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:28.896 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:28.669+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30218 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:28.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:28.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:28.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:28.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:28.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:28.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:28.897 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:28.546+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:29.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:28.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:29.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:28.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:29.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:28.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:29.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:28.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:29.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:29.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:29.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:29.200+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:29.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:29.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:29.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:29.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:29.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:29.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:29.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:29.531+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:30.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:29.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:30.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:29.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:30.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:29.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:30.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:29.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:30.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:30.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:30.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:30.170+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:30.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:30.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:30.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:30.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:30.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:30.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:30.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:30.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:31.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:30.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:31.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:30.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:31.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:30.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:31.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:30.923+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:31.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:31.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:31.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:31.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:31.211+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:31.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:31.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:31.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:31.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:31.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:31.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:31.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:31.508+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:31.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:31.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:31.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:31.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:31.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:31.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:31.946+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:32.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:32.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:32.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:32.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:32.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:32.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:32.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:32.259+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:32.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:32.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:32.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:32.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:32.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:32.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:32.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:32.537+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:32.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:32.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:32.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:32.931+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:33.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:33.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:33.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:33.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:33.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:33.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:33.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:33.491+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:33.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:33.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:33.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:33.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:33.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:33.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:33.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:33.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:33.904 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:33.673+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30307 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:33.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:33.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:33.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:33.903+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:34.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:34.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:34.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:34.292+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:34.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:34.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:34.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:34.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:34.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:34.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:34.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:34.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:35.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:35.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:35.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:35.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:35.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:35.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:35.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:35.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:36.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:35.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:36.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:35.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:36.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:35.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:36.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:35.824+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:36.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:36.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:36.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:36.318+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:36.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:36.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:36.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:36.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:36.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:36.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:36.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:36.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:36.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:36.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:36.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:36.785+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:37.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:37.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:37.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:37.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:37.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:37.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:37.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:37.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:37.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:37.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:37.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:37.748+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:38.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:38.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:38.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:38.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:38.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:38.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:38.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:38.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:38.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:38.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:38.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:38.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:38.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:38.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:38.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:38.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:38.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:38.676+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30397 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:38.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:38.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:38.709+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:38.710+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:39.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:39.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:39.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:39.447+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:39.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:39.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:39.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:39.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:40.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:39.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:40.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:39.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:40.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:39.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:40.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:39.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:40.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:40.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:40.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:40.416+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:40.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:40.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:40.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:40.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:40.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:40.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:40.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:40.777+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:41.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:41.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:41.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:41.267+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:41.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:41.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:41.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:41.466+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:41.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:41.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:41.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:41.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:42.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:42.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:42.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:42.471+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:42.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:42.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:42.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:42.233+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:42.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:42.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:42.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:42.786+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:43.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:43.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:43.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:43.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:43.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:43.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:43.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:43.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:43.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:43.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:43.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:43.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:43.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:43.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:43.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:43.247+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:43.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:43.679+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30486 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:43.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:43.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:43.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:43.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:44.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:44.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:44.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:44.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:44.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:44.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:44.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:44.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:44.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:44.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:44.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:44.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:45.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:45.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:45.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:45.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:45.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:45.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:45.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:45.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:45.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:45.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:45.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:45.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:46.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:46.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:46.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:46.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:46.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:46.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:46.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:46.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:46.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:46.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:46.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:46.769+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:47.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:47.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:47.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:47.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:47.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:47.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:47.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:47.353+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:47.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:47.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:47.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:47.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:48.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:48.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:48.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:48.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:48.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:48.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:48.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:48.379+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:48.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:48.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:48.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:48.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:48.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:48.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:48.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:48.311+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:48.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:48.683+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30576 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:48.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:48.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:48.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:48.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:49.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:49.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:49.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:49.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:49.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:49.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:49.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:49.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:49.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:49.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:49.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:49.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:49.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:49.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:49.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:49.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:49.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:49.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:49.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:49.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:50.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:50.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:50.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:50.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:50.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:50.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:50.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:50.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:50.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:50.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:50.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:50.667+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:51.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:51.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:51.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:51.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:51.294+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:51.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:51.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:51.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:51.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:51.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:51.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:51.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:51.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:51.644+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:52.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:52.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:52.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:52.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:52.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:52.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:52.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:52.451+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:52.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:52.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:52.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:52.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:52.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:52.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:52.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:52.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:52.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:52.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:52.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:52.619+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:53.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:53.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:53.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:53.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:53.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:53.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:53.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:53.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:53.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:53.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:53.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:53.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:53.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:53.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:53.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:53.306+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:53.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:53.686+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30665 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:53.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:53.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:53.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:53.607+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:54.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:54.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:54.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:54.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:54.524+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:54.560 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:54.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:54.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:54.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:54.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:54.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:54.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:54.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:54.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:54.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:54.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:54.641+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:55.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:55.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:55.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:55.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:55.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:55.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:55.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:55.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:55.598 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:55.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:55.598 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:55.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:55.598 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:55.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:55.598 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:55.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:55.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:55.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:55.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:55.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:56.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:56.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:56.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:56.501+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:56.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:56.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:56.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:56.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:56.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:56.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:56.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:56.366+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:57.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:57.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:57.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:57.519+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:57.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:57.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:57.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:57.413+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:57.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:57.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:57.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:57.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:57.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:58.706 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:31:58.690+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30754 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:31:58.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:58.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:58.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:58.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:58.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:58.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:58.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:58.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:58.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:58.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:58.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:58.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:58.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:58.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:58.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:58.444+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:58.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:58.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:58.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:58.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:58.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:58.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:58.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:58.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:59.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:59.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:59.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:31:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:31:59.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:59.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:59.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:59.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:31:59.516+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:59.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:59.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:59.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:31:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:31:59.472+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:00.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:00.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:00.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:00.556+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:00.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:00.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:00.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:00.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:00.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:00.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:00.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:00.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:00.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:00.510+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:01.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:01.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:01.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:01.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:01.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:01.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:01.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:01.624+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:01.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:01.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:01.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:01.536+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:02.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:02.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:02.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:02.566+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:02.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:02.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:02.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:02.614+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:02.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:02.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:02.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:02.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:02.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:02.576+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:03.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:03.693+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30843 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:03.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:03.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:03.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:03.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:03.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:03.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:03.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:03.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:03.823 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:03.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:03.823 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:03.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:03.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:03.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:03.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:03.591+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:03.824 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:03.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:03.824 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:03.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:03.824 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:03.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:03.824 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:03.560+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:04.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:04.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:04.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:04.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:04.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:04.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:04.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:04.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:04.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:04.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:04.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:04.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:04.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:04.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:04.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:05.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:05.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:05.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:05.555+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:05.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:05.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:05.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:05.525+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:05.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:05.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:05.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:05.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:05.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:05.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:06.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:06.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:06.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:06.577+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:06.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:06.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:06.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:06.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:06.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:06.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:06.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:06.477+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:06.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:06.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:06.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:06.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:06.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:06.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:06.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:07.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:07.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:07.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:07.614+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:07.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:07.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:07.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:07.557+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:07.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:07.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:07.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:07.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:07.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:08.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:08.698+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30932 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:08.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:08.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:08.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:08.591+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:08.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:08.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:08.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:08.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:08.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:08.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:08.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:08.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:09.593+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:09.593+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:09.593+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:09.594+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:09.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:09.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:09.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:09.556+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:09.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:09.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:09.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:09.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:10.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:10.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:10.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:10.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:10.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:10.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:10.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:10.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:10.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:10.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:10.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:10.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:10.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:10.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:11.750 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:11.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:11.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:11.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:11.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:11.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:11.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:11.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:11.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:11.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:11.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:11.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:11.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:11.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:11.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:11.541+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:11.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:11.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:11.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:11.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:11.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:11.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:11.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:11.499+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:12.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:12.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:12.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:12.520+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:12.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:12.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:12.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:12.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:12.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:12.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:12.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:12.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:12.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:12.500+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:13.731 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:13.701+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31021 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:13.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:13.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:13.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:13.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:13.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:13.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:13.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:13.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:13.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:13.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:13.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:13.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:13.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:13.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:13.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:13.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:13.732 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:13.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:13.733 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:13.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:13.733 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:13.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:13.733 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:13.471+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:14.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:14.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:14.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:14.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:14.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:14.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:14.580+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:14.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:14.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:14.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:14.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:14.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:14.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:14.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:14.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:14.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:14.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:14.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:14.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:14.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:14.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:14.977 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:14.484+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:15.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:15.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:15.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:15.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:15.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:15.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:15.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:15.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:15.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:15.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:15.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:15.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:16.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:16.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:16.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:16.633+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:16.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:16.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:16.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:16.536+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:16.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:16.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:16.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:16.519+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:17.649+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:17.649+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:17.649+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:17.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:17.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:17.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:17.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:17.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:17.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:17.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:17.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:17.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:17.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:17.545+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:18.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:18.705+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31111 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:18.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:18.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:18.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:18.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:18.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:18.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:18.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:18.607+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:18.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:18.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:18.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:18.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:18.833 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:18.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:18.833 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:18.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:18.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:18.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:18.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:18.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:18.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:18.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:18.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:18.504+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:19.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:19.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:19.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:19.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:19.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:19.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:19.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:19.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:19.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:19.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:19.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:19.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:19.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:19.552+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:20.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:20.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:20.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:20.595+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:20.523+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:20.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:20.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:20.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:20.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:20.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:20.558+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:21.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:21.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:21.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:21.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:21.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:21.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:21.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:21.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:21.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:21.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:21.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:21.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:22.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:22.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:22.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:22.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:22.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:22.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:22.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:22.592+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:22.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:22.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:22.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:22.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:22.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:22.568+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:23.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:23.709+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31199 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:23.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:23.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:23.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:23.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:23.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:23.576+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:23.576+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:23.576+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:23.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:23.577+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:23.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:23.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:23.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:23.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:23.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:23.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:23.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:23.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:24.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:24.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:24.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:24.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:24.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:24.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:24.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:24.605+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:24.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:24.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:24.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:24.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:24.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:24.612+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:25.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:25.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:25.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:25.621+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:25.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:25.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:25.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:25.588+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:25.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:25.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:25.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:25.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:25.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:25.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:25.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:26.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:26.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:26.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:26.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:26.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:26.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:26.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:26.636+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:26.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:26.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:26.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:26.583+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:27.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:27.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:27.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:27.549+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:27.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:27.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:27.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:27.629+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:27.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:27.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:27.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:27.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:27.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:27.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:27.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:27.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:28.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:28.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:28.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:28.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:28.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:28.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:28.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:28.630+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:28.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:28.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:28.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:28.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:28.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:28.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:28.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:28.581+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:28.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:28.713+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31290 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:28.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:28.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:28.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:28.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:28.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:28.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:28.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:28.531+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:29.576+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:29.576+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:29.576+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:29.577+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:29.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:29.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:29.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:29.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:29.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:29.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:29.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:29.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:29.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:29.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:29.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:29.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:30.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:30.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:30.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:30.624+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:30.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:30.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:30.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:30.678+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:30.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:30.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:30.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:30.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:30.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:30.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:30.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:31.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:31.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:31.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:31.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:31.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:31.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:31.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:31.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:31.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:31.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:31.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:31.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:31.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:31.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:31.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:31.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:31.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:31.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:31.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:31.686+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:32.570+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:32.570+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:32.570+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:32.571+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:32.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:32.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:32.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:32.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:32.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:32.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:32.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:32.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:33.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:33.716+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31375 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:33.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:33.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:33.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:33.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:33.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:33.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:33.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:33.584+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:33.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:33.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:33.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:33.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:33.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:33.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:33.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:33.673+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:33.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:33.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:33.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:33.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:33.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:33.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:33.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:33.541+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:34.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:34.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:34.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:34.625+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:34.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:34.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:34.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:34.630+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:34.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:34.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:34.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:34.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:34.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:34.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:34.541+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:35.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:35.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:35.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:35.606+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:35.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:35.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:35.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:35.655+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:35.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:35.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:35.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:35.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:35.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:35.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:35.526+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:36.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:36.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:36.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:36.561+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:36.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:36.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:36.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:36.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:36.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:36.687+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:36.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:36.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:36.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:36.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:36.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:36.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:36.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:36.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:37.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:37.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:37.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:37.496+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:37.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:37.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:37.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:37.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:37.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:37.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:37.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:37.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:37.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:37.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:37.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:37.659+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:38.746 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:38.720+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31465 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:38.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:38.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:38.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:38.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:38.747 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:38.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:38.747 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:38.598+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:38.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:38.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:38.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:38.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:38.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:38.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:38.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:38.609+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:38.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:38.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:38.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:38.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:38.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:38.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:38.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:38.490+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:39.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:39.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:39.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:39.561+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:39.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:39.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:39.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:39.583+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:39.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:39.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:39.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:39.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:39.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:39.516+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:39.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:39.517+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:40.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:40.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:40.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:40.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:40.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:40.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:40.579+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:40.580+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:40.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:40.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:40.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:40.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:40.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:40.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:40.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:40.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:41.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:41.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:41.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:41.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:41.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:41.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:41.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:41.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:41.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:41.588+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:41.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:41.588+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:41.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:41.588+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:41.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:41.589+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:41.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:41.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:41.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:41.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:41.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:41.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:41.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:41.490+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:42.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:42.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:42.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:42.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:42.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:42.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:42.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:42.445+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:42.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:42.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:42.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:42.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:42.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:42.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:42.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:42.645+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:43.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:43.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:43.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:43.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:43.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:43.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:43.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:43.586+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:43.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:43.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:43.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:43.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:43.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:43.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:43.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:43.427+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:43.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:43.724+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31554 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:43.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:43.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:43.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:43.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:44.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:44.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:44.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:44.633+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:44.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:44.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:44.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:44.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:44.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:44.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:44.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:44.399+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:45.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:45.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:45.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:45.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:45.611+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:45.611+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:45.611+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:45.612+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:45.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:45.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:45.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:45.430+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:46.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:46.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:46.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:46.646+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:46.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:46.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:46.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:46.598+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:46.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:46.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:46.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:46.468+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:47.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:47.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:47.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:47.499+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:47.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:47.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:47.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:47.643+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:47.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:47.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:47.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:47.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:47.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:47.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:47.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:47.627+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:48.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:48.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:48.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:48.669+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:48.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:48.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:48.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:48.457+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:48.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:48.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:48.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:48.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:48.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:48.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:48.646+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:49.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:48.728+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31643 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:49.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:49.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:49.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:49.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:49.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:49.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:49.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:49.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:49.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:49.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:49.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:49.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:49.673+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:49.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:49.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:49.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:49.674+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:50.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:50.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:50.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:50.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:50.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:50.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:50.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:50.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:50.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:50.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:50.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:50.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:51.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:51.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:51.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:51.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:51.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:51.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:51.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:51.648+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:51.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:51.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:51.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:51.728+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:52.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:52.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:52.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:52.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:52.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:52.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:52.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:52.402+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:52.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:52.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:52.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:52.682+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:52.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:52.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:52.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:52.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:52.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:52.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:52.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:52.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:53.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:53.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:53.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:53.655+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:53.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:53.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:53.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:53.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:53.732+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31733 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:53.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:53.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:53.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:53.736+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:54.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:54.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:54.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:54.663+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:54.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:54.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:54.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:54.453+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:54.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:54.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:54.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:54.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:55.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:55.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:55.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:55.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:55.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:55.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:55.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:55.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:55.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:55.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:55.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:55.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:55.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:55.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:55.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:55.427+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:55.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:55.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:55.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:55.706+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:56.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:56.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:56.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:56.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:56.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:56.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:56.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:56.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:56.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:56.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:56.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:56.683+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:57.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:57.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:57.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:57.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:57.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:57.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:57.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:57.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:57.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:57.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:57.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:57.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:58.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:58.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:58.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:58.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:58.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:58.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:58.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:58.439+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:32:58.736+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31818 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:32:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:58.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:58.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:58.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:32:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:58.764+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:59.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:59.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:59.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:32:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:32:59.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:32:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:59.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:59.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:59.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:32:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:32:59.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:00.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:59.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:00.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:59.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:00.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:59.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:00.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:32:59.773+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:00.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:00.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:00.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:00.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:00.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:00.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:00.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:00.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:00.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:00.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:00.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:00.753+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:01.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:01.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:01.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:01.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:01.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:01.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:01.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:01.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:01.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:01.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:01.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:01.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:02.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:02.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:02.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:02.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:02.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:02.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:02.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:02.682+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:02.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:02.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:02.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:02.691+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:03.643 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:03.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:03.643 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:03.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:03.643 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:03.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:03.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:03.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:03.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:03.739+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31900 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:03.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:03.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:03.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:03.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:03.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:03.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:03.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:03.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:04.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:04.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:04.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:04.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:04.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:04.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:04.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:04.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:04.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:04.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:04.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:04.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:04.692+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:04.692+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:04.692+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:04.693+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:05.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:05.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:05.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:05.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:05.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:05.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:05.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:05.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:05.652+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:05.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:05.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:05.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:05.729+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:06.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:06.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:06.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:06.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:06.615+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:06.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:06.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:06.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:06.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:06.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:06.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:06.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:06.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:06.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:07.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:07.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:07.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:07.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:07.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:07.584+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:07.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:07.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:07.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:07.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:07.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:07.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:07.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:07.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:07.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:07.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:08.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:08.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:08.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:08.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:08.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:08.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:08.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:08.382+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:09.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:08.743+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 31978 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:08.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:08.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:09.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:08.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:09.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:08.738+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:09.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:09.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:09.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:09.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:09.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:09.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:09.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:09.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:09.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:09.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:09.667+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:09.668+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:09.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:09.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:09.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:09.725+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:10.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:10.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:10.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:10.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:10.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:10.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:10.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:10.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:10.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:10.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:10.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:10.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:11.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:11.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:11.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:11.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:11.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:11.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:11.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:11.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:11.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:11.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:11.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:11.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:11.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:11.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:11.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:11.732+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:12.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:12.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:12.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:12.661+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:12.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:12.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:12.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:12.691+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:12.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:12.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:12.462+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:12.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:13.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:13.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:13.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:13.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:13.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:13.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:13.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:13.446+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:14.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:13.748+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32060 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:13.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:13.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:14.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:13.727+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:14.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:13.727+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:14.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:14.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:14.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:14.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:14.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:14.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:14.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:14.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:14.736+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:15.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:15.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:15.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:15.412+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:15.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:15.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:15.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:15.646+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:15.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:15.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:15.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:15.745+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:16.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:16.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:16.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:16.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:16.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:16.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:16.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:16.599+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:16.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:16.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:16.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:16.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:16.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:16.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:16.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:16.747+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:17.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:17.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:17.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:17.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:17.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:17.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:17.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:17.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:17.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:17.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:17.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:17.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:17.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:17.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:17.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:17.753+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:18.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:18.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:18.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:18.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:18.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:18.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:18.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:18.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:18.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:18.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:18.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:18.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:19.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:18.752+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32142 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:18.770+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:18.770+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:18.770+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:18.771+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:19.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:19.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:19.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:19.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:19.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:19.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:19.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:19.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:19.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:19.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:20.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:19.819+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:20.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:19.819+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:20.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:19.819+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:20.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:19.820+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:20.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:20.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:20.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:20.410+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:20.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:20.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:20.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:20.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:20.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:20.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:20.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:20.535+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:20.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:20.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:20.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:20.859+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:21.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:21.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:21.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:21.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:21.504+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:21.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:21.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:21.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:21.445+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:21.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:21.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:21.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:21.833+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:22.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:22.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:22.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:22.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:22.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:22.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:22.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:22.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:22.453+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:22.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:22.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:22.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:22.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:23.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:23.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:23.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:23.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:23.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:23.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:23.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:23.495+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:23.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:23.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:23.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:23.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:23.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:23.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:23.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:23.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:24.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:23.755+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32229 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:23.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:23.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:23.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:23.832+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:24.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:24.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:24.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:24.430+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:24.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:24.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:24.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:24.538+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:24.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:24.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:24.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:24.832+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:25.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:25.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:25.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:25.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:25.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:25.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:25.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:25.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:25.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:25.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:25.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:25.857+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:26.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:26.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:26.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:26.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:26.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:26.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:26.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:26.509+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:26.873 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:26.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:26.873 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:26.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:26.873 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:26.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:26.874 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:26.500+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:26.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:26.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:26.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:26.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:27.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:27.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:27.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:27.457+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:27.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:27.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:27.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:27.498+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:27.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:27.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:27.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:27.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:28.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:28.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:28.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:28.419+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:28.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:28.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:28.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:28.513+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:29.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:28.759+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32317 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:28.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:28.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:28.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:28.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:29.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:29.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:29.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:29.453+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:29.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:29.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:29.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:29.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:30.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:29.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:30.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:29.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:30.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:29.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:30.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:29.812+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:30.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:30.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:30.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:30.412+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:30.836 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:30.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:30.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:30.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:30.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:30.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:30.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:30.503+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:30.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:30.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:30.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:30.834+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:31.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:31.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:31.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:31.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:31.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:31.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:31.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:31.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:31.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:31.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:31.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:31.885+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:32.864 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:32.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:32.864 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:32.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:32.864 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:32.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:32.864 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:32.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:32.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:32.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:32.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:32.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:32.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:32.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:32.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:32.480+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:32.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:32.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:32.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:32.897+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:33.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:33.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:33.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:33.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:33.511+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:33.511+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:33.511+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:33.511+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:34.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:33.763+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32408 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:33.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:33.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:33.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:33.880+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:34.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:34.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:34.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:34.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:34.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:34.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:34.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:34.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:34.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:34.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:34.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:34.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:34.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:34.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:34.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:34.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:34.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:34.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:34.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:34.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:35.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:35.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:35.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:35.428+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:35.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:35.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:35.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:35.450+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:35.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:35.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:35.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:35.897+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:36.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:36.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:36.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:36.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:36.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:36.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:36.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:36.467+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:36.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:36.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:36.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:36.849+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:37.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:37.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:37.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:37.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:37.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:37.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:37.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:37.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:37.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:37.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:37.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:37.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:37.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:37.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:37.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:37.472+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:37.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:37.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:37.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:37.825+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:38.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:38.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:38.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:38.514+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:38.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:38.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:38.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:38.448+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:39.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:38.766+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32496 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:38.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:38.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:38.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:38.795+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:39.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:39.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:39.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:39.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:39.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:39.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:39.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:39.488+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:40.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:39.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:40.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:39.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:40.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:39.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:40.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:39.827+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:40.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:40.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:40.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:40.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:40.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:40.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:40.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:40.475+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:40.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:40.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:40.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:40.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:40.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:40.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:40.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:40.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:40.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:40.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:40.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:40.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:41.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:41.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:41.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:41.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:41.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:41.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:41.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:41.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:42.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:41.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:42.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:41.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:42.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:41.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:42.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:41.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:42.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:42.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:42.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:42.445+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:42.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:42.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:42.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:42.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:42.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:42.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:42.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:42.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:43.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:43.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:43.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:43.480+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:43.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:43.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:43.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:43.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:44.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:43.771+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32586 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:43.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:43.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:43.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:43.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:44.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:44.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:44.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:44.523+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:44.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:44.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:44.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:44.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:45.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:44.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:45.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:44.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:45.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:44.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:45.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:44.879+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:45.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:45.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:45.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:45.484+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:45.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:45.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:45.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:45.466+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:45.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:45.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:45.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:45.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:46.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:46.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:46.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:46.448+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:46.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:46.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:46.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:46.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:46.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:46.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:46.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:46.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:47.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:47.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:47.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:47.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:47.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:47.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:47.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:47.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:47.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:47.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:47.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:47.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:48.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:48.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:48.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:48.518+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:48.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:48.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:48.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:48.412+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:49.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:48.774+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32674 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:48.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:49.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:48.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:49.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:48.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:49.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:48.824+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:49.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:49.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:49.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:49.514+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:49.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:49.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:49.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:49.432+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:50.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:49.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:50.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:49.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:50.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:49.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:50.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:49.804+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:50.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:50.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:50.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:50.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:50.512+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:50.512+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:50.512+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:50.512+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:50.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:50.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:50.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:50.756+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:51.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:51.378+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:51.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:51.378+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:51.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:51.378+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:51.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:51.379+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:51.805 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:51.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:51.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:51.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:51.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:51.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:51.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:51.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:51.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:51.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:51.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:51.803+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:52.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:52.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:52.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:52.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:52.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:52.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:52.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:52.353+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:52.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:52.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:52.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:52.768+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:53.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:53.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:53.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:53.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:53.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:53.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:53.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:53.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:53.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:53.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:53.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:53.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:53.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:53.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:53.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:53.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:53.905 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:53.779+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32765 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:53.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:53.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:53.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:53.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:53.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:53.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:53.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:53.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:54.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:54.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:54.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:54.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:54.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:54.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:54.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:54.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:54.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:54.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:54.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:54.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:55.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:55.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:55.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:55.546+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:55.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:55.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:55.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:55.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:55.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:55.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:55.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:55.775+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:56.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:56.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:56.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:56.347+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:56.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:56.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:56.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:56.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:56.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:56.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:56.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:56.766+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:57.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:57.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:57.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:57.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:57.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:57.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:57.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:57.540+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:57.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:57.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:57.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:57.741+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:58.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:58.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:58.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:58.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:58.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:58.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:58.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:58.537+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:58.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:58.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:58.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:58.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:58.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:58.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:58.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:58.315+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:58.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:33:58.782+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32853 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:33:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:58.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:58.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:58.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:58.699+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:59.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:59.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:59.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:59.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:59.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:59.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:33:59.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:33:59.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:33:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:59.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:59.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:59.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:33:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:33:59.677+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:59.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:59.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:59.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:33:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:33:59.587+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:00.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:00.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:00.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:00.315+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:00.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:00.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:00.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:00.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:00.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:00.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:00.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:00.715+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:01.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:01.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:01.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:01.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:01.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:01.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:01.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:01.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:01.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:01.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:01.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:01.738+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:01.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:01.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:01.559+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:01.560+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:02.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:02.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:02.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:02.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:02.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:02.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:02.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:02.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:02.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:02.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:02.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:02.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:02.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:02.739+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:03.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:03.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:03.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:03.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:03.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:03.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:03.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:03.523+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:03.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:03.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:03.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:03.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:03.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:03.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:03.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:03.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:03.917 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:03.787+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 32943 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:03.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:03.711+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:03.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:03.711+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:03.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:03.711+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:03.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:03.712+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:04.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:04.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:04.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:04.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:04.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:04.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:04.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:04.552+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:04.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:04.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:04.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:04.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:04.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:04.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:04.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:04.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:04.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:04.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:04.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:04.685+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:05.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:05.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:05.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:05.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:05.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:05.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:05.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:05.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:05.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:05.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:05.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:05.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:05.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:05.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:05.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:05.645+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:06.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:06.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:06.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:06.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:06.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:06.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:06.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:06.559+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:06.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:06.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:06.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:06.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:06.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:06.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:06.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:06.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:06.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:06.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:06.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:06.652+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:07.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:07.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:07.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:07.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:07.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:07.515+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:07.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:07.516+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:07.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:07.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:07.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:07.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:07.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:07.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:07.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:07.288+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:07.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:07.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:07.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:07.685+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:08.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:08.512+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:08.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:08.512+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:08.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:08.512+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:08.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:08.513+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:08.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:08.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:08.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:08.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:08.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:08.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:08.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:08.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:08.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:08.791+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33031 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:08.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:08.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:08.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:08.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:09.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:09.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:09.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:09.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:09.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:09.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:09.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:09.515+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:09.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:09.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:09.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:09.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:09.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:09.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:09.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:09.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:09.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:09.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:09.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:09.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:10.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:10.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:10.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:10.294+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:10.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:10.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:10.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:10.564+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:10.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:10.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:10.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:10.748+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:11.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:11.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:11.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:11.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:11.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:11.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:11.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:11.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:11.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:11.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:11.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:11.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:11.588+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:11.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:11.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:11.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:11.787+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:12.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:12.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:12.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:12.552+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:12.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:12.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:12.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:12.328+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:12.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:12.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:12.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:12.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:13.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:13.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:13.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:13.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:13.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:13.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:13.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:13.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:13.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:13.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:13.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:13.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:13.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:13.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:13.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:13.349+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:13.923 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:13.795+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33122 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:13.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:13.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:13.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:13.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:14.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:14.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:14.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:14.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:14.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:14.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:14.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:14.596+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:14.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:14.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:14.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:14.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:14.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:14.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:14.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:14.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:14.675+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:14.675+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:14.675+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:14.676+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:15.667 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:15.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:15.667 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:15.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:15.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:15.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:15.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:15.564+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:15.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:15.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:15.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:15.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:15.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:15.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:15.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:15.665+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:16.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:16.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:16.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:16.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:16.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:16.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:16.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:16.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:16.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:16.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:16.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:16.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:16.617+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:16.617+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:16.617+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:16.618+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:17.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:17.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:17.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:17.248+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:17.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:17.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:17.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:17.606+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:17.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:17.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:17.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:17.600+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:18.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:18.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:18.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:18.200+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:18.925 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:18.799+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33210 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:18.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:18.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:18.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:18.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:18.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:18.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:18.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:18.606+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:18.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:18.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:18.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:18.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:18.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:18.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:18.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:18.638+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:19.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:19.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:19.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:19.168+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:19.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:19.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:19.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:19.640+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:19.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:19.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:19.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:19.640+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:20.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:20.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:20.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:20.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:20.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:20.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:20.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:20.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:20.683+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:20.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:20.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:20.643+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:20.644+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:21.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:21.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:21.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:21.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:21.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:21.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:21.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:21.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:21.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:21.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:21.681+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:21.681+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:22.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:22.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:22.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:22.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:22.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:22.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:22.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:22.643+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:22.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:22.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:22.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:22.689+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:23.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:23.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:23.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:23.151+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:23.927 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:23.803+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33300 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:23.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:23.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:23.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:23.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:23.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:23.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:23.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:23.650+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:23.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:23.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:23.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:23.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:23.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:23.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:23.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:23.615+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:24.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:24.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:24.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:24.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:24.566+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:24.566+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:24.566+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:24.567+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:24.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:24.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:24.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:24.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:24.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:24.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:24.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:24.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:25.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:25.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:25.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:25.141+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:25.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:25.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:25.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:25.637+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:25.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:25.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:25.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:25.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:26.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:26.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:26.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:26.095+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:26.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:26.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:26.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:26.661+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:26.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:26.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:26.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:26.629+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:27.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:27.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:27.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:27.077+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:27.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:27.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:27.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:27.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:27.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:27.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:27.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:27.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:28.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:28.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:28.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:28.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:28.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:28.807+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33388 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:28.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:28.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:28.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:28.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:28.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:28.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:28.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:28.580+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:28.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:28.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:28.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:28.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:28.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:28.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:28.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:28.589+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:29.155+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:29.155+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:29.155+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:29.156+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:29.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:29.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:29.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:29.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:29.596+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:29.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:29.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:29.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:29.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:30.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:30.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:30.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:30.126+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:30.620 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-20T08:34:30.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-20T08:34:30.625 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-20T08:34:30.626 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-20T08:34:30.638 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-20T08:34:30.648 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task sleep... 2024-02-20T08:34:30.649 INFO:teuthology.task.sleep:Sleeping for 3600 seconds 2024-02-20T08:34:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:30.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:30.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:30.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:30.599+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:30.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:30.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:30.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:30.582+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:31.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:31.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:31.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:31.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:31.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:31.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:31.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:31.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:31.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:31.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:31.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:31.627+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:32.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:32.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:32.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:32.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:32.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:32.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:32.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:32.588+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:32.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:32.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:32.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:32.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:33.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:33.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:33.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:33.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:33.938 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:33.811+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33479 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:33.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:33.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:33.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:33.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:33.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:33.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:33.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:33.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:33.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:33.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:33.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:33.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:33.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:33.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:33.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:33.619+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:34.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:34.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:34.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:34.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:34.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:34.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:34.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:34.582+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:34.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:34.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:34.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:34.665+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:35.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:35.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:35.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:35.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:35.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:35.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:35.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:35.655+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:35.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:35.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:35.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:35.575+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:36.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:36.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:36.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:36.168+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:36.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:36.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:36.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:36.615+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:36.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:36.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:36.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:36.623+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:37.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:37.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:37.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:37.196+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:37.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:37.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:37.645+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:37.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:37.646+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:37.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:37.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:37.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:37.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:38.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:38.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:38.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:38.184+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:38.942 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:38.815+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33567 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:38.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:38.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:38.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:38.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:38.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:38.626+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:38.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:38.627+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:38.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:38.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:38.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:38.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:38.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:38.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:38.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:38.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:39.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:39.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:39.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:39.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:39.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:39.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:39.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:39.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:39.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:39.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:39.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:39.641+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:40.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:40.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:40.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:40.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:40.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:40.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:40.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:40.181+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:40.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:40.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:40.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:40.606+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:40.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:40.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:40.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:40.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:41.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:41.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:41.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:41.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:41.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:41.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:41.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:41.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:41.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:41.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:41.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:41.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:41.594+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:42.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:42.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:42.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:42.131+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:42.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:42.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:42.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:42.575+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:42.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:42.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:42.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:42.603+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:43.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:43.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:43.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:43.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:43.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:43.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:43.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:43.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:43.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:43.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:43.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:43.593+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:43.823 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:43.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:43.823 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:43.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:43.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:43.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:43.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:43.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:44.098 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:43.820+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33658 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:44.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:44.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:44.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:44.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:44.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:44.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:44.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:44.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:44.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:44.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:44.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:44.575+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:45.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:45.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:45.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:45.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:45.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:45.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:45.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:45.625+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:45.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:45.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:45.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:45.621+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:46.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:46.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:46.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:46.135+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:46.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:46.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:46.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:46.614+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:46.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:46.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:46.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:46.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:46.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:46.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:46.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:46.615+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:47.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:47.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:47.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:47.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:47.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:47.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:47.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:47.584+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:47.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:47.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:47.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:47.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:48.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:48.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:48.147+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:48.148+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:48.949 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:48.823+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33747 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:48.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:48.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:48.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:48.612+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:48.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:48.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:48.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:48.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:48.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:48.596+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:49.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:49.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:49.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:49.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:49.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:49.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:49.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:49.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:49.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:49.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:49.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:49.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:49.633+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:50.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:50.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:50.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:50.179+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:50.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:50.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:50.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:50.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:50.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:50.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:50.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:50.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:51.183+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:51.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:51.183+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:51.183+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:51.183+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:51.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:51.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:51.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:51.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:51.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:51.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:51.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:51.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:51.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:51.527+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:52.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:52.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:52.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:52.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:52.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:52.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:52.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:52.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:52.511+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:52.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:52.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:52.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:52.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:53.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:53.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:53.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:53.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:53.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:53.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:53.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:53.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:53.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:53.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:53.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:53.577+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:53.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:53.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:53.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:53.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:53.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:53.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:53.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:53.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:54.147 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:53.828+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33839 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:54.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:54.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:54.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:54.145+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:54.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:54.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:54.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:54.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:54.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:54.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:54.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:54.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:54.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:55.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:55.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:55.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:55.162+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:55.808 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:55.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:55.808 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:55.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:55.808 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:55.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:55.808 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:55.598+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:55.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:55.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:55.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:55.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:55.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:55.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:55.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:55.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:56.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:56.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:56.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:56.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:56.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:56.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:56.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:56.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:56.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:56.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:56.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:56.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:56.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:56.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:56.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:56.485+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:57.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:57.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:57.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:57.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:57.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:57.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:57.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:57.580+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:57.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:57.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:57.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:57.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:57.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:57.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:57.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:57.500+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:58.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:58.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:58.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:58.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:58.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:58.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:58.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:58.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:58.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:58.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:58.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:58.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:58.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:58.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:58.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:58.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:58.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:58.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:58.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:58.496+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:59.194 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:34:58.832+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 33928 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:34:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:59.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:59.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:59.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:34:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:34:59.192+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:59.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:59.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:59.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:34:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:34:59.630+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:34:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:59.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:59.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:59.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:34:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:34:59.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:00.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:00.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:00.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:00.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:00.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:00.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:00.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:00.195+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:00.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:00.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:00.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:00.585+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:00.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:00.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:00.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:00.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:01.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:01.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:01.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:01.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:01.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:01.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:01.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:01.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:01.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:01.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:01.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:01.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:01.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:01.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:01.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:02.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:02.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:02.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:02.217+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:02.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:02.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:02.591+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:02.592+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:02.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:02.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:02.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:02.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:02.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:03.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:03.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:03.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:03.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:03.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:03.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:03.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:03.249+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:03.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:03.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:03.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:03.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:03.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:03.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:03.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:03.616+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:03.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:03.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:03.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:03.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:03.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:03.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:03.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:03.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:04.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:03.836+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34019 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:04.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:04.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:04.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:04.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:04.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:04.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:04.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:04.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:04.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:04.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:04.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:04.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:04.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:04.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:04.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:04.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:04.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:04.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:04.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:04.665+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:05.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:05.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:05.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:05.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:05.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:05.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:05.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:05.519+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:05.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:05.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:05.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:05.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:05.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:05.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:05.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:05.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:05.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:05.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:05.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:05.630+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:06.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:06.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:06.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:06.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:06.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:06.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:06.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:06.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:06.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:06.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:06.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:06.563+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:06.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:06.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:06.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:06.621+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:07.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:07.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:07.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:07.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:07.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:07.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:07.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:07.267+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:07.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:07.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:07.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:07.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:07.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:07.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:07.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:07.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:08.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:08.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:08.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:08.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:08.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:08.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:08.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:08.230+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:08.866 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:08.840+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34108 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:08.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:08.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:08.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:08.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:08.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:08.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:08.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:08.620+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:08.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:08.612+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:08.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:08.612+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:08.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:08.612+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:08.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:08.613+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:09.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:09.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:09.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:09.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:09.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:09.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:09.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:09.221+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:09.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:09.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:09.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:09.667+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:09.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:09.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:09.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:09.645+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:10.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:10.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:10.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:10.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:10.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:10.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:10.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:10.681+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:10.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:10.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:10.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:10.639+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:11.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:11.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:11.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:11.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:11.739 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:11.675+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:11.739 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:11.675+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:11.739 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:11.675+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:11.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:11.675+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:11.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:11.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:11.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:11.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:11.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:11.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:11.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:11.677+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:12.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:12.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:12.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:12.162+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:12.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:12.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:12.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:12.680+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:12.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:12.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:12.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:12.708+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:13.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:13.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:13.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:13.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:13.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:13.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:13.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:13.967 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:13.845+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34200 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:13.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:13.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:13.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:13.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:13.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:13.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:13.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:13.691+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:13.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:13.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:13.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:13.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:13.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:13.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:13.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:13.720+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:14.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:14.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:14.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:14.156+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:14.707+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:14.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:14.707+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:14.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:14.707+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:14.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:14.708+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:14.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:14.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:14.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:14.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:14.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:15.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:15.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:15.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:15.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:15.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:15.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:15.718+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:15.719+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:15.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:15.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:15.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:15.733+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:16.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:16.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:16.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:16.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:16.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:16.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:16.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:16.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:16.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:16.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:16.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:16.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:16.699+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:17.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:17.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:17.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:17.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:17.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:17.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:17.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:17.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:17.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:17.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:17.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:17.707+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:18.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:18.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:18.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:18.080+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:18.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:18.849+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34289 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:18.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:18.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:18.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:18.778+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:18.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:18.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:18.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:18.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:19.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:19.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:19.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:19.130+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:19.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:19.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:19.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:19.807+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:19.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:19.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:19.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:19.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:20.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:20.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:20.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:20.167+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:20.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:20.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:20.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:20.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:20.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:20.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:20.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:20.692+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:21.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:20.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:21.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:20.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:21.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:20.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:21.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:20.796+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:21.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:21.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:21.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:21.142+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:22.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:21.838+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:22.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:21.838+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:22.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:21.838+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:22.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:21.839+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:22.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:21.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:22.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:21.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:22.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:21.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:22.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:21.741+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:22.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:22.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:22.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:22.185+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:22.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:22.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:22.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:22.714+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:22.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:22.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:22.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:22.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:22.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:22.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:22.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:22.855+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:23.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:23.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:23.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:23.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:23.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:23.853+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34380 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:23.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:23.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:23.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:23.748+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:23.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:23.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:23.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:23.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:23.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:23.878+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:24.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:24.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:24.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:24.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:24.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:24.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:25.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:24.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:24.862+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:24.751+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:24.751+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:24.751+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:24.752+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:25.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:25.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:25.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:25.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:25.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:25.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:25.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:26.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:25.723+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:25.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:25.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:25.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:25.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:26.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:26.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:26.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:26.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:26.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:26.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:26.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:26.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:26.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:26.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:26.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:26.707+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:27.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:26.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:27.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:26.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:27.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:26.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:27.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:26.859+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:27.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:27.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:27.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:27.200+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:27.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:27.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:27.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:27.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:27.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:28.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:27.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:28.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:27.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:28.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:27.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:28.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:27.888+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:28.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:28.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:28.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:28.217+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:29.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:28.857+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34469 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:28.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:28.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:28.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:28.737+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:28.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:28.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:28.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:29.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:28.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:29.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:29.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:29.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:29.221+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:30.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:29.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:30.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:29.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:30.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:29.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:30.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:29.753+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:30.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:29.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:30.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:29.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:30.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:29.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:30.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:29.939+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:30.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:30.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:30.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:30.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:30.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:30.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:30.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:30.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:31.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:30.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:31.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:30.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:30.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:31.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:30.773+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:31.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:31.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:31.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:31.217+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:31.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:31.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:31.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:31.980+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:31.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:31.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:31.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:31.776+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:32.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:32.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:32.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:32.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:32.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:32.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:32.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:32.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:32.761+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:32.761+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:32.761+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:32.761+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:33.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:33.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:33.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:33.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.024+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:33.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:33.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:33.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:33.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:33.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:33.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:33.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:33.311+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:33.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:33.862+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34561 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:33.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:33.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:33.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:33.737+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:34.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:34.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:34.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:34.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:33.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:34.264+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:34.264+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:34.264+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:34.265+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:35.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:34.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:35.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:34.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:35.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:34.732+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:35.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:34.733+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:35.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:35.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:35.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:35.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:35.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:35.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:35.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:35.217+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:36.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:35.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:36.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:35.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:36.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:35.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:36.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:35.721+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:36.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:36.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:36.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:36.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:36.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:36.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:36.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:36.072+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:36.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:36.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:36.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:36.686+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:37.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:37.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:37.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:37.191+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:37.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:37.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:37.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:37.075+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:37.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:37.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:37.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:37.686+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:38.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:38.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:38.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:38.198+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:38.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:38.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:38.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:38.126+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:38.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:38.866+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34650 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:38.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:38.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:38.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:38.693+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:39.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:39.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:39.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:39.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:39.192+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:39.192+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:39.192+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:39.192+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:39.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:39.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:39.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:39.665+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:40.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:40.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:40.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:40.198+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:40.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:40.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:40.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:40.146+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:40.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:40.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:40.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:40.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:41.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:41.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:41.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:41.224+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:41.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:41.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:41.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:41.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:41.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:41.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:41.689+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:41.690+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:42.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:42.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:42.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:42.244+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:42.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:42.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:42.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:42.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:42.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:42.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:42.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:42.658+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:43.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:43.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:43.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:43.164+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:43.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:43.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:43.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:43.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:43.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:43.871+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34741 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:43.705+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:43.705+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:43.705+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:43.706+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:44.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:44.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:44.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:44.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:44.159+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:44.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:44.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:44.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:44.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:44.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:44.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:44.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:44.326+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:44.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:44.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:44.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:44.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:44.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:44.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:44.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:44.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:45.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:45.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:45.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:45.135+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:45.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:45.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:45.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:45.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:45.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:45.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:45.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:45.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:45.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:45.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:45.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:45.685+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:46.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:46.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:46.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:46.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:46.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:46.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:46.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:46.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:46.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:46.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:46.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:46.669+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:47.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:47.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:47.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:47.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:47.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:47.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:47.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:47.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:47.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:47.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:47.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:47.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:47.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:47.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:47.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:47.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:47.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:47.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:47.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:47.664+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:48.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:48.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:48.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:48.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:48.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:48.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:48.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:48.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:48.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:48.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:48.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:48.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:48.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:48.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:48.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:48.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:49.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:48.875+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34830 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:49.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:49.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:49.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:49.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:49.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:49.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:49.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:49.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:49.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:49.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:49.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:49.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:49.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:49.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:49.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:49.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:49.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:49.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:49.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:49.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:50.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:50.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:50.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:50.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:50.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:50.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:50.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:50.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:50.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:50.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:50.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:50.570+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:51.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:51.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:51.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:51.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:50.988+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:51.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:51.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:51.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:51.354+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:51.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:51.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:51.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:51.568+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:52.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:52.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:52.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:52.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:52.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:52.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:52.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:52.578+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:52.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:52.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:52.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:52.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:52.389+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:53.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:53.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:53.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:53.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:53.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:53.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:53.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:53.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:53.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:53.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:53.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:53.614+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:53.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:53.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:53.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:53.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:53.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:53.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:53.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:53.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:54.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:53.879+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 34922 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:54.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:54.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:54.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:54.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:54.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:54.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:54.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:54.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:54.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:54.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:54.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:54.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:54.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:54.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:54.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:54.652+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:55.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:55.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:55.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:55.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.010+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:55.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:55.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:55.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:55.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:55.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:55.388+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:55.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:55.389+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:55.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:55.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:55.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:55.693+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:56.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:55.984+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:56.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:56.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:56.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:56.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:56.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:56.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:56.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:56.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:56.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:56.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:56.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:56.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:56.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:56.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:56.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:56.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:56.968+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:57.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:57.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:57.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:57.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:57.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:57.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:57.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:57.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:57.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:57.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:57.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:57.616+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:58.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:58.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:58.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:58.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.012+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:58.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:58.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:58.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:58.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:58.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:58.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:58.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:58.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:35:58.884+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35011 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:35:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:35:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:58.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:59.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:59.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:59.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:35:59.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:59.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:59.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:59.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:35:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:35:59.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:00.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:59.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:00.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:59.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:00.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:59.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:00.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:35:59.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:00.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:00.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:00.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:00.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:00.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:00.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:00.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:00.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:00.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:00.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:00.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:00.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:00.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:00.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:00.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:00.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:00.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:00.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:00.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:00.960+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:01.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:01.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:01.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:01.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:01.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:01.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:01.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:01.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:01.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:01.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:01.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:01.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:01.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:01.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:01.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:01.535+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:01.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:01.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:01.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:01.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:02.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:02.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:02.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:02.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:02.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:02.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:02.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:02.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:02.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:02.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:02.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:02.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:02.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:02.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:03.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:03.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:03.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:03.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:03.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:03.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:03.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:03.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:03.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:03.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:03.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:03.557+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:03.911 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:03.888+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35102 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:03.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:03.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:03.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:03.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:04.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:04.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:04.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:04.514+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:04.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:04.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:04.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:04.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:04.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:04.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:04.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:04.988+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:05.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:05.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:05.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:05.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:05.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:05.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:05.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:05.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:05.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:06.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:05.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:06.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:05.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:06.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:05.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:06.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:05.978+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:06.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:06.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:06.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:06.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:06.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:06.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:06.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:06.510+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:07.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:07.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:07.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:07.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:07.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:07.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:07.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:07.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:07.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:07.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:07.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:07.478+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:08.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:08.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:08.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:08.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:08.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:08.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:08.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:08.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:08.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:08.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:08.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:08.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:09.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:08.892+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35191 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:09.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:09.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:09.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:09.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:09.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:09.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:09.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:09.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:09.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:09.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:09.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:09.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:10.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:10.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:10.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:10.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:10.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:10.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:10.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:10.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:10.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:10.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:10.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:10.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:10.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:10.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:10.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:10.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:11.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:11.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:11.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:11.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:11.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:11.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:11.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:11.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:11.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:11.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:11.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:11.419+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:11.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:11.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:11.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:11.630+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:12.419 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:12.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:12.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:12.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:12.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:12.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:12.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:12.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:12.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:12.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:12.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:12.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:12.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:12.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:12.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:12.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:13.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:13.087+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:13.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:13.087+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:13.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:13.087+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:13.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:13.088+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:13.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:13.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:13.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:13.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:13.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:13.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:13.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:13.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:13.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:13.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:13.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:13.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:13.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:13.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:13.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:13.591+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:13.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:13.897+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35282 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:14.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:14.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:14.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:14.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:14.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:14.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:14.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:14.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:14.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:14.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:14.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:14.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:14.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:14.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:14.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:14.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:15.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:15.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:15.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:15.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:15.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:15.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:15.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:15.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:15.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:15.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:15.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:15.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:15.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:15.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:15.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:15.357+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:15.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:15.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:15.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:15.527+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:16.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:16.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:16.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:16.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:16.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:16.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:16.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:16.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:16.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:16.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:16.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:16.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:16.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:16.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:16.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:16.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:17.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:17.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:17.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:17.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:17.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:17.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:17.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:17.125+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:17.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:17.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:17.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:17.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:18.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:18.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:18.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:18.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:18.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:18.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:18.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:18.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:18.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:18.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:18.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:18.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:18.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:18.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:18.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:18.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:18.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:18.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:18.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:18.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:18.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:18.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:18.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:18.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:19.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:18.901+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35372 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:19.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:19.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:19.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:19.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:19.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:19.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:19.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:19.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:19.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:19.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:19.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:19.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:20.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:20.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:20.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:20.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:20.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:20.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:20.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:20.171+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:20.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:20.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:20.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:20.584+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:21.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:21.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:21.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:21.412+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:21.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:21.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:21.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:21.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:21.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:21.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:21.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:21.633+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:22.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:22.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:22.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:22.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:22.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:22.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:22.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:22.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:22.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:22.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:22.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:22.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:22.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:22.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:22.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:22.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:22.659+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:23.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:23.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:23.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:23.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:23.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:23.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:23.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:23.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:23.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:23.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:23.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:23.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:23.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:23.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:23.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:23.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:23.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:23.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:23.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:23.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:23.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:23.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:23.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:23.906+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35461 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:23.698+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:24.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:24.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:24.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:24.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:24.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:24.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:24.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:24.094+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:24.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:24.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:24.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:24.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:25.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:24.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:25.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:24.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:25.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:24.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:25.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:24.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:25.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:25.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:25.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:25.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:25.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:25.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:25.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:25.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:25.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:25.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:25.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:25.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:26.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:25.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:26.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:25.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:26.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:25.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:26.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:25.766+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:26.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:26.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:26.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:26.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:26.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:26.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:26.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:26.386+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:27.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:26.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:27.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:26.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:27.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:26.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:27.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:26.745+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:27.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:27.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:27.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:27.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:27.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:27.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:27.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:27.175+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:27.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:27.344+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:27.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:27.344+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:27.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:27.344+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:27.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:27.345+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:27.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:27.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:27.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:27.735+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:28.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:28.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:28.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:28.185+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:28.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:28.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:28.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:28.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:29.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:28.910+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35551 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:28.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:28.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:28.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:28.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:29.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:29.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:29.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:29.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:29.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:29.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:29.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:29.219+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:29.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:29.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:29.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:29.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:30.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:29.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:30.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:29.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:30.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:29.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:30.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:29.773+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:30.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:30.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:30.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:30.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:30.180+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:30.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:30.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:30.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:30.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:30.379+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:30.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:30.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:30.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:30.795+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:31.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:31.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:31.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:31.371+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:31.219+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:31.219+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:31.219+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:31.220+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:31.820+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:31.820+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:31.820+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:31.821+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:32.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:32.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:32.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:32.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:32.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:32.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:32.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:32.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:33.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:32.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:33.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:32.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:33.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:32.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:33.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:32.802+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:33.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:33.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:33.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:33.432+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:33.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:33.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:33.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:33.183+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:33.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:33.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:33.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:33.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:33.776+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:34.200 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:33.913+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35642 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:34.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:34.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:34.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:34.197+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:34.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:34.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:34.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:34.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:35.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:34.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:35.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:34.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:35.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:34.807+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:35.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:34.808+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:35.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:35.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:35.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:35.195+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:35.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:35.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:35.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:35.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:35.385+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:36.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:35.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:36.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:35.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:36.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:35.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:36.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:35.763+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:36.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:36.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:36.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:36.209+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:36.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:36.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:36.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:36.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:36.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:36.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:36.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:36.772+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:37.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:37.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:37.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:37.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:37.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:37.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:37.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:37.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:37.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:37.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:37.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:37.789+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:38.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:38.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:38.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:38.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:38.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:38.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:38.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:38.416+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:39.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:38.919+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35731 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:38.798+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:38.798+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:38.798+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:38.799+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:39.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:39.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:39.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:39.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:39.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:39.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:39.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:39.240+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:39.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:39.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:40.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:39.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:40.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:39.768+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:40.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:40.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:40.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:40.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:40.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:40.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:40.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:40.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:40.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:40.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:40.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:40.741+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:41.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:41.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:41.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:41.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:41.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:41.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:41.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:41.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:41.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:41.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:41.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:41.782+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:42.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:42.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:42.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:42.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:42.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:42.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:42.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:42.264+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:42.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:42.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:42.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:42.744+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:43.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:43.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:43.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:43.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:43.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:43.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:43.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:43.412+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:43.659 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:43.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:43.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:43.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:43.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:43.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:43.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:43.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:43.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:43.779+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:43.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:43.779+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:43.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:43.779+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:43.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:43.780+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:44.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:43.924+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35820 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:44.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:44.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:44.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:44.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:44.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:44.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:44.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:44.309+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:44.822+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:44.822+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:44.822+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:44.823+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:45.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:45.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:45.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:45.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:45.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:45.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:45.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:45.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:45.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:45.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:45.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:45.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:45.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:46.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:46.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:46.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:46.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:46.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:46.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:46.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:46.283+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:46.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:46.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:46.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:46.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:47.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:47.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:47.411+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:47.412+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:47.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:47.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:47.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:47.328+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:47.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:47.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:47.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:47.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:48.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:48.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:48.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:48.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:48.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:48.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:48.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:48.379+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:48.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:48.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:48.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:48.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:48.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:48.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:48.928+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35910 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:48.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:48.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:48.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:48.717+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:49.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:49.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:49.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:49.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:49.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:49.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:49.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:49.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:49.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:49.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:49.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:49.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:50.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:50.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:50.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:50.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:50.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:50.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:50.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:50.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:50.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:50.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:50.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:50.742+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:51.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:51.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:51.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:51.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:51.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:51.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:51.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:51.405+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:51.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:51.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:51.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:51.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:51.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:51.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:51.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:51.306+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:51.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:51.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:51.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:51.706+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:52.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:52.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:52.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:52.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:52.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:52.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:52.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:52.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:52.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:52.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:52.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:52.736+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:53.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:53.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:53.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:53.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:53.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:53.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:53.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:53.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:53.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:53.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:53.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:53.314+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:53.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:53.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:53.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:53.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:53.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:53.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:53.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:53.695+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:53.933+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 35999 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:54.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:54.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:54.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:54.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:54.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:54.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:54.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:54.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:54.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:54.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:54.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:54.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:54.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:54.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:54.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:54.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:54.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:54.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:54.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:54.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:54.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:54.659+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:55.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:55.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:55.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:55.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:55.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:55.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:55.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:55.428+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:55.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:55.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:55.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:55.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:55.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:55.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:55.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:55.398+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:55.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:55.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:55.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:55.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:55.660+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:56.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:56.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:56.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:56.638+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:56.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:56.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:56.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:56.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:56.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:56.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:56.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:56.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:56.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:56.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:56.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:56.441+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:57.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:57.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:57.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:57.640+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:57.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:57.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:57.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:57.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:57.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:57.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:57.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:57.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:57.414+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:58.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:58.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:58.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:58.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:58.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:58.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:58.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:58.423+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:58.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:58.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:58.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:58.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:58.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:58.671+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:58.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:58.672+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:59.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:36:58.937+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36088 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:36:59.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:59.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:59.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:59.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:59.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:59.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:36:59.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:36:59.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:59.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:59.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:59.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:59.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:59.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:59.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:36:59.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:36:59.390+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:36:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:59.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:59.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:59.673+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:36:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:36:59.673+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:00.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:00.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:00.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:00.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:00.411+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:00.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:00.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:00.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:00.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:00.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:00.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:00.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:00.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:00.671+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:01.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:01.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:01.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:01.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:01.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:01.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:01.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:01.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:01.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:01.340+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:01.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:01.340+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:01.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:01.340+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:01.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:01.341+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:01.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:01.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:01.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:01.692+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:02.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:02.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:02.457+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:02.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:02.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:02.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:02.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:02.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:02.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:02.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:02.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:02.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:02.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:03.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:03.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:03.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:03.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:03.446+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:03.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:03.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:03.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:03.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:03.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:03.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:03.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:03.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:03.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:03.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:03.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:03.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:03.688+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:04.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:03.942+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36178 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:04.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:04.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:04.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:04.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:04.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:04.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:04.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:04.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:04.702 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:04.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:04.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:04.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:04.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:04.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:04.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:04.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:04.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:04.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:04.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:04.699+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:05.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:05.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:05.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:05.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:05.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:05.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:05.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:05.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:05.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:05.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:05.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:05.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:05.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:05.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:05.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:05.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:05.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:05.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:05.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:05.694+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:06.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:06.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:06.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:06.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:06.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:06.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:06.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:06.424+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:06.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:06.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:06.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:06.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:06.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:06.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:06.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:06.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:06.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:06.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:06.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:06.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:07.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:07.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:07.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:07.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:07.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:07.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:07.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:07.387+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:07.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:07.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:07.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:07.721+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:08.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:08.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:08.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:08.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:08.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:08.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:08.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:08.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:08.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:08.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:08.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:08.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:08.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:08.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:08.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:08.366+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:08.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:08.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:08.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:08.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:08.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:08.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:08.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:08.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:09.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:08.946+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36268 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:09.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:09.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:09.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:09.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:09.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:09.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:09.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:09.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:09.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:09.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:09.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:09.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:09.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:09.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:09.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:09.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:09.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:09.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:09.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:09.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:10.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:10.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:10.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:10.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:10.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:10.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:10.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:10.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:10.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:10.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:10.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:10.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:10.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:10.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:10.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:10.363+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:10.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:10.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:10.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:10.593+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:11.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:11.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:11.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:11.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:11.371+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:11.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:11.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:11.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:11.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:11.613+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:11.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:11.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:11.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:11.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:12.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:12.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:12.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:12.431+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:12.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:12.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:12.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:12.571+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:12.363+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:12.363+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:12.363+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:12.364+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:13.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:13.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:13.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:13.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:13.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:13.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:13.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:13.559+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:13.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:13.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:13.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:13.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:13.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:13.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:13.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:13.445+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:13.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:13.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:13.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:13.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:13.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:14.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:13.951+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36360 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:14.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:14.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:14.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:14.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:14.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:14.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:14.588+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:14.588+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:14.588+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:14.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:14.589+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:14.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:14.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:14.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:14.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:15.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:15.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:15.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:15.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:15.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:15.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:15.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:15.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:15.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:15.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:15.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:15.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:15.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:15.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:15.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:15.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:15.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:15.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:16.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:16.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:16.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:16.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:16.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:16.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:16.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:16.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:16.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:16.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:16.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:16.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:16.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:16.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:16.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:16.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:16.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:16.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:16.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:16.577+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:17.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:17.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:17.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:17.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:17.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:17.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:17.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:17.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:17.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:17.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:17.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:17.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:17.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:17.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:17.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:17.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:17.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:18.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:18.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:18.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:18.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:18.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:18.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:18.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:18.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:18.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:18.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:18.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:18.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:18.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:18.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:18.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:18.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:18.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:18.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:19.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:18.955+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36450 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:19.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:19.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:19.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:19.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:19.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:19.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:19.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:19.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:19.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:19.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:19.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:19.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:19.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:19.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:19.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:19.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:19.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:19.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:19.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:19.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:20.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:20.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:20.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:20.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:20.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:20.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:20.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:20.443+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:20.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:20.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:20.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:20.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:20.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:20.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:20.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:20.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:20.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:20.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:20.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:20.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:21.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:21.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:21.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:21.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:21.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:21.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:21.460+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:21.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:21.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:21.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:21.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:21.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:21.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:21.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:21.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:21.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:22.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:22.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:22.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:22.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:22.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:22.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:22.478+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:22.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:22.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:22.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:22.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:22.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:22.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:22.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:22.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:22.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:22.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:22.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:22.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:22.362+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:23.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:23.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:23.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:23.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:23.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:23.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:23.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:23.493+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:23.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:23.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:23.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:23.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:23.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:23.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:23.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:23.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:23.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:23.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:23.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:23.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:23.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:23.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:23.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:23.578+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:24.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:23.959+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36539 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:24.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:24.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:24.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:24.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:24.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:24.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:24.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:24.564+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:24.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:24.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:24.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:24.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:24.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:24.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:24.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:24.506+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:25.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:25.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:25.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:25.562+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:25.538+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:25.538+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:25.538+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:25.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:25.539+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:25.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:25.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:25.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:25.368+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:26.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:26.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:26.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:26.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:26.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:26.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:26.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:26.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:26.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:26.579+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:26.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:26.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:26.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:26.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:26.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:26.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:26.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:26.414+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:27.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:27.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:27.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:27.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:27.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:27.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:27.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:27.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:27.582+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:27.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:27.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:27.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:27.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:28.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:28.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:28.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:28.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:28.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:28.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:28.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:28.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:28.570+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:28.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:28.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:28.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:28.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:29.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:28.964+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36629 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:29.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:29.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:29.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:29.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:29.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:29.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:29.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:29.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:29.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:29.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:29.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:29.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:29.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:29.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:29.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:29.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:29.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:29.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:29.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:29.379+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:30.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:30.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:30.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:30.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:30.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:30.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:30.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:30.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:30.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:30.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:30.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:30.580+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:30.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:30.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:30.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:30.495+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:31.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:31.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:31.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:31.630+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:31.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:31.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:31.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:31.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:31.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:31.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:31.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:31.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:31.432+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:32.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:32.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:32.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:32.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:32.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:32.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:32.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:32.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:32.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:32.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:32.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:32.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:32.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:32.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:33.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:33.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:33.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:33.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:33.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:33.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:33.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:33.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:33.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:33.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:33.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:33.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:33.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:33.427+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:34.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:33.969+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36719 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:34.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:34.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:34.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:34.611+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:34.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:34.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:34.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:34.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:34.546+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:34.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:34.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:34.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:34.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:35.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:35.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:35.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:35.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:35.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:35.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:35.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:35.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:35.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:35.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:35.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:35.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:35.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:35.440+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:36.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:36.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:36.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:36.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:36.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:36.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:36.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:36.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:36.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:36.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:36.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:36.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:36.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:37.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:37.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:37.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:37.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:37.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:37.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:37.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:37.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:37.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:37.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:37.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:37.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:37.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:37.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:37.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:37.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:37.644+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:38.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:38.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:38.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:38.629+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:38.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:38.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:38.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:38.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:38.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:38.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:38.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:38.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:38.416+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:39.393 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:38.973+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36808 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:39.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:39.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:39.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:39.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:39.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:39.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:39.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:39.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:39.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:39.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:39.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:39.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:40.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:40.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:40.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:40.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:40.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:40.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:40.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:40.428+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:40.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:40.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:40.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:40.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:40.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:40.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:40.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:40.583+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:41.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:41.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:41.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:41.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:41.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:41.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:41.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:41.601+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:41.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:41.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:41.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:41.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:41.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:41.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:41.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:41.663+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:41.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:41.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:41.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:41.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:41.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:41.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:41.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:41.420+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:42.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:42.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:42.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:42.665+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:42.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:42.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:42.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:42.594+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:42.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:42.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:42.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:42.428+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:43.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:43.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:43.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:43.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:43.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:43.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:43.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:43.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:43.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:43.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:43.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:43.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:43.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:43.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:43.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:43.419+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:43.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:43.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:43.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:43.698+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:44.387 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:43.978+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36898 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:44.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:44.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:44.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:44.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:44.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:44.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:44.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:44.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:44.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:44.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:44.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:44.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:44.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:44.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:44.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:44.385+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:44.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:44.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:44.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:44.710+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:45.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:45.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:45.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:45.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:45.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:45.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:45.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:45.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:45.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:45.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:45.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:45.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:45.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:45.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:45.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:45.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:45.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:45.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:45.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:45.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:46.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:46.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:46.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:46.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:46.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:46.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:46.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:46.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:46.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:46.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:46.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:46.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:46.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:46.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:46.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:46.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:46.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:46.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:46.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:46.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:47.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:47.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:47.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:47.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:47.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:47.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:47.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:47.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:47.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:47.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:47.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:47.719+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:48.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:48.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:48.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:48.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:48.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:48.583+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:48.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:48.584+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:48.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:48.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:48.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:48.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:48.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:48.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:48.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:48.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:48.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:48.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:48.709+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:48.710+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:49.402 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:48.982+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 36987 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:49.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:49.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:49.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:49.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:49.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:49.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:49.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:49.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:49.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:49.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:49.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:49.723+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:50.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:50.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:50.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:50.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:50.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:50.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:50.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:50.444+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:50.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:50.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:50.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:50.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:50.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:50.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:50.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:50.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:50.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:50.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:50.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:50.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:50.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:50.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:50.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:50.734+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:51.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:51.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:51.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:51.402+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:51.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:51.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:51.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:51.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:51.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:51.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:51.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:51.742+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:52.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:52.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:52.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:52.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:52.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:52.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:52.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:52.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:52.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:52.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:52.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:52.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:52.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:52.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:52.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:52.355+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:52.715+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:52.715+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:52.715+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:52.716+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:53.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:53.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:53.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:53.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:53.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:53.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:53.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:53.586+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:53.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:53.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:53.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:53.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:53.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:53.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:53.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:53.345+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:53.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:53.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:53.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:53.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:54.378 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:53.987+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37077 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:54.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:54.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:54.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:54.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:54.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:54.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:54.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:54.580+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:54.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:54.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:54.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:54.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:54.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:54.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:54.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:54.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:54.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:54.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:54.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:54.665+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:55.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:55.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:55.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:55.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:55.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:55.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:55.650 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:55.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:55.650 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:55.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:55.650 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:55.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:55.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:55.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:55.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:55.646+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:56.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:56.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:56.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:56.428+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:56.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:56.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:56.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:56.610+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:56.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:56.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:56.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:56.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:56.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:57.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:57.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:57.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:57.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:57.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:57.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:57.657+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:57.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:57.657+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:57.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:57.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:57.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:57.456+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:58.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:58.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:58.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:58.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:58.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:58.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:58.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:58.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:58.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:58.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:58.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:58.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:58.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:58.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:58.443+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:58.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:58.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:58.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:58.698+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:59.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:37:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:37:58.991+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37166 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:37:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:59.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:59.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:59.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:37:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:37:59.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:37:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:59.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:59.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:59.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:59.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:59.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:37:59.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:37:59.697+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:59.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:59.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:59.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:59.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:59.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:59.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:37:59.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:37:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:37:59.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:00.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:00.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:00.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:00.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:00.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:00.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:00.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:00.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:00.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:00.683+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:00.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:00.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:00.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:00.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:00.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:00.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:00.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:00.515+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:01.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:01.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:01.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:01.607+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:01.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:01.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:01.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:01.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:01.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:01.634+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:01.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:01.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:01.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:01.561+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:02.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:02.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:02.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:02.519+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:02.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:02.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:02.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:02.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:02.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:02.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:02.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:02.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:02.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:02.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:02.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:02.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:02.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:02.596+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:03.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:03.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:03.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:03.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:03.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:03.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:03.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:03.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:03.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:03.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:03.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:03.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:03.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:03.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:03.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:03.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:03.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:03.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:04.486 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:03.996+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37256 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:04.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:04.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:04.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:04.613+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:04.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:04.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:04.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:04.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:04.641+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:04.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:04.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:04.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:04.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:04.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:04.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:04.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:04.598+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:05.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:05.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:05.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:05.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:05.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:05.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:05.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:05.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:05.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:05.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:05.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:05.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:05.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:05.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:05.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:05.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:06.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:06.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:06.667+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:06.668+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:06.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:06.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:06.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:06.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:06.681+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:06.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:06.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:06.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:06.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:06.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:06.557+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:06.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:06.558+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:07.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:07.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:07.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:07.638+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:07.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:07.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:07.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:07.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:07.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:07.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:07.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:07.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:07.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:07.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:07.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:07.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:07.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:07.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:07.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:08.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:08.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:08.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:08.593+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:08.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:08.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:08.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:08.692+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:08.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:08.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:08.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:08.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:08.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:08.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:08.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:08.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:09.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:09.001+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37346 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:09.642+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:09.642+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:09.642+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:09.643+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:09.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:09.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:09.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:09.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:09.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:09.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:09.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:09.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:09.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:09.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:09.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:09.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:10.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:10.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:10.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:10.670+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:10.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:10.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:10.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:10.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:10.658+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:10.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:10.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:10.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:10.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:10.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:10.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:10.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:10.623+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:11.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:11.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:11.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:11.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:11.649+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:11.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:11.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:11.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:11.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:11.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:11.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:11.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:11.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:11.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:11.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:11.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:11.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:11.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:11.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:11.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:12.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:12.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:12.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:12.658+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:12.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:12.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:12.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:12.647+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:12.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:12.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:12.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:12.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:13.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:13.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:13.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:13.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:13.652+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:13.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:13.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:13.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:13.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:13.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:13.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:13.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:13.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:13.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:13.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:13.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:13.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:13.637+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:14.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:14.005+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37437 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:14.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:14.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:14.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:14.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:14.661+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:14.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:14.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:14.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:14.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:14.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:14.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:14.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:14.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:14.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:14.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:14.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:14.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:15.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:15.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:15.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:15.572+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:15.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:15.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:15.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:15.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:15.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:15.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:15.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:15.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:15.631+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:16.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:16.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:16.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:16.606+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:16.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:16.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:16.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:16.611+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:16.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:16.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:16.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:16.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:16.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:16.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:16.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:16.664+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:17.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:17.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:17.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:17.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:17.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:17.638+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:17.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:17.707+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:17.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:17.707+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:17.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:17.707+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:17.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:17.708+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:17.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:17.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:17.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:17.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:17.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:17.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:17.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:17.659+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:18.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:18.623+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:18.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:18.625+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:18.758+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:18.758+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:18.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:18.758+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:18.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:18.759+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:18.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:18.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:18.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:18.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:18.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:18.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:18.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:18.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:19.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:19.010+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37527 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:19.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:19.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:19.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:19.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:19.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:19.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:19.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:19.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:19.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:19.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:19.719+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:19.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:19.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:19.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:19.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:19.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:19.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:19.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:19.666+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:20.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:20.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:20.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:20.671+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:20.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:20.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:20.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:20.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:20.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:20.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:20.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:20.727+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:20.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:20.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:20.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:20.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:20.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:20.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:20.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:20.632+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:21.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:21.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:21.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:21.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:21.753+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:21.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:21.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:21.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:21.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:21.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:21.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:21.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:21.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:21.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:21.660+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:22.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:22.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:22.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:22.703+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:22.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:22.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:22.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:22.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:22.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:22.768+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:22.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:22.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:22.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:22.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:22.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:22.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:22.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:22.582+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:23.692+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:23.692+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:23.692+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:23.693+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:23.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:23.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:23.738+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:23.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:23.739+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:23.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:23.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:23.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:23.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:23.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:23.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:23.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:23.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:24.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:24.015+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37617 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:24.512+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:24.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:24.513+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:24.513+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:24.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:24.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:24.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:24.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:24.719+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:24.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:24.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:24.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:24.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:24.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:24.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:24.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:24.669+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:25.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:25.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:25.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:25.730+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:25.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:25.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:25.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:25.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:25.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:25.666+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:25.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:25.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:25.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:25.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:25.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:25.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:25.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:25.543+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:26.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:26.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:26.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:26.734+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:26.701+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:26.701+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:26.701+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:26.702+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:26.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:26.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:26.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:26.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:26.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:26.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:26.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:26.583+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:27.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:27.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:27.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:27.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:27.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:27.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:27.783 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:27.684+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:27.783 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:27.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:27.783 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:27.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:27.783 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:27.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:27.783 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:27.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:27.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:27.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:27.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:27.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:28.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:28.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:28.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:28.654+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:28.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:28.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:28.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:28.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:28.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:28.764+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:28.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:28.764+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:28.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:28.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:28.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:28.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:28.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:28.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:28.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:28.694+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:29.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:29.019+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37707 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:29.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:30.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:29.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:30.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:29.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:30.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:29.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:30.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:29.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:30.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:29.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:30.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:29.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:30.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:29.722+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:30.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:29.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:30.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:29.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:30.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:29.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:30.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:29.609+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:30.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:30.708+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:30.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:30.708+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:30.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:30.708+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:30.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:30.708+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:30.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:30.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:30.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:30.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:30.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:30.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:30.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:30.737+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:30.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:30.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:30.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:30.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:30.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:30.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:30.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:30.589+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:31.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:31.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:31.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:31.776+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:31.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:31.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:31.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:31.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:31.680+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:31.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:31.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:31.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:31.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:32.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:32.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:32.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:32.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:32.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:32.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:32.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:32.658+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:32.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:32.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:32.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:32.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:33.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:33.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:33.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:33.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:33.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:33.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:33.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:33.795+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:33.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:33.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:33.922 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:33.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:33.922 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:33.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:33.922 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:33.630+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:33.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:33.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:33.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:33.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:33.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:33.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:33.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:33.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:34.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:34.024+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37798 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:34.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:34.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:34.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:34.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:34.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:34.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:34.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:34.652+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:34.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:34.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:34.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:34.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:34.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:34.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:34.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:34.553+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:34.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:34.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:34.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:34.836+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:35.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:35.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:35.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:35.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:35.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:35.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:35.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:35.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:35.862 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:35.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:35.862 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:35.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:35.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:35.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:35.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:35.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:35.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:35.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:35.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:35.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:36.876 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:36.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:36.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:36.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:36.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:36.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:36.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:36.595+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:36.877 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:36.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:36.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:36.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:36.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:36.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:36.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:36.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:36.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:36.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:36.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:36.874+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:37.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:37.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:37.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:37.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:37.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:37.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:37.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:37.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:37.865 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:37.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:37.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:37.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:37.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:37.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:37.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:37.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:37.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:37.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:37.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:37.862+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:38.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:38.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:38.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:38.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:38.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:38.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:38.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:38.570+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:38.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:38.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:38.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:38.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:38.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:38.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:38.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:38.672+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:39.155 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:39.028+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37887 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:39.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:38.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:39.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:38.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:39.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:38.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:39.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:38.845+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:39.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:39.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:39.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:39.611+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:39.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:39.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:39.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:39.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:39.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:39.799+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:39.799+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:39.799+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:39.800+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:40.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:40.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:40.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:40.832+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:40.832+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:40.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:40.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:40.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:40.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:40.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:40.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:40.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:40.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:40.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:40.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:40.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:40.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:40.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:40.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:41.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:41.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:41.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:41.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:41.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:41.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:41.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:41.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:41.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:41.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:41.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:41.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:41.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:41.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:41.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:41.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:41.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:41.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:41.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:41.861+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:42.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:42.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:42.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:42.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:42.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:42.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:42.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:42.595+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:42.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:42.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:42.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:42.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:42.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:42.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:42.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:42.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:42.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:42.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:42.907+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:42.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:43.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:43.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:43.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:43.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:43.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:43.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:43.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:43.612+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:43.879 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:43.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:43.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:43.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:43.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:43.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:43.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:43.738+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:44.157 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:44.033+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 37977 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:44.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:43.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:44.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:43.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:44.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:43.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:44.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:43.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:44.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:44.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:44.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:44.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:44.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:44.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:44.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:44.611+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:44.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:44.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:44.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:44.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:44.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:44.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:44.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:44.712+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:44.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:44.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:44.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:44.892+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:45.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:45.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:45.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:45.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:45.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:45.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:45.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:45.759+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:45.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:45.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:45.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:45.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:45.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:45.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:45.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:45.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:46.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:45.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:46.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:45.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:46.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:45.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:46.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:45.869+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:46.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:46.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:46.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:46.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:46.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:46.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:46.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:46.778+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:46.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:46.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:46.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:46.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:46.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:46.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:46.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:46.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:46.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:46.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:46.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:46.893+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:47.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:47.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:47.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:47.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:47.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:47.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:47.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:47.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:47.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:47.749+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:47.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:47.749+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:47.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:47.749+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:47.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:47.750+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:47.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:47.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:47.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:47.893+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:48.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:48.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:48.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:48.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:48.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:48.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:48.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:48.556+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:48.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:48.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:48.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:48.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:48.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:48.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:48.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:48.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:49.166 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:49.037+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38067 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:49.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:48.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:49.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:48.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:49.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:48.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:49.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:48.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:49.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:49.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:49.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:49.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:49.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:49.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:49.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:49.590+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:49.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:49.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:49.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:49.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:49.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:49.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:49.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:49.768+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:49.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:49.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:49.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:49.902+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:50.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:50.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:50.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:50.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:50.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:50.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:50.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:50.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:50.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:50.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:50.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:50.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:50.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:50.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:50.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:50.769+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:50.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:50.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:50.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:50.949+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:51.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:51.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:51.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:51.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:51.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:51.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:51.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:51.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:51.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:51.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:51.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:51.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:51.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:51.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:52.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:51.991+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:52.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:51.991+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:52.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:51.991+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:52.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:51.991+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:52.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:52.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:52.754+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:52.755+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:52.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:52.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:52.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:52.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:52.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:52.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:52.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:52.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:52.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:52.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:52.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:52.972+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:53.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:53.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:53.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:53.778+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:53.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:53.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:53.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:53.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:53.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:53.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:53.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:53.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:54.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:54.042+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38158 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:53.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:53.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:53.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:53.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:54.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:54.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:54.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:54.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:54.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:54.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:54.969 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:54.772+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:54.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:54.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:54.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:54.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:54.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:54.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:54.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:54.610+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:54.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:54.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:54.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:54.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:55.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:55.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:55.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:55.641+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:55.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:55.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:55.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:55.810+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:55.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:55.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:55.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:55.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:56.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:56.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:56.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:56.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:56.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:56.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:56.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:56.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:56.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:56.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:56.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:56.977+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:57.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:57.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:57.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:57.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:57.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:57.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:57.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:57.692+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:57.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:57.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:57.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:57.798+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:58.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:58.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:58.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:58.013+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:58.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:58.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:58.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:58.659+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:58.848+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:58.848+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:58.848+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:58.849+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:59.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:38:59.047+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38248 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:38:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:59.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:59.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:59.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:38:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:38:59.002+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:59.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:59.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:59.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:38:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:38:59.820+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:38:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:59.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:59.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:59.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:38:59.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:38:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:38:59.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.008+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:00.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:00.978 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:00.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:00.978 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:00.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:00.978 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:00.671+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:00.978 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:00.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:00.978 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:00.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:00.978 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:00.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:00.979 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:00.796+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:00.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:01.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:01.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:01.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:01.680+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:01.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:01.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:01.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:01.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:01.787+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:02.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:02.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:02.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:02.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:02.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:02.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:02.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:02.751+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:02.970+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:03.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:03.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:03.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:03.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:03.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:03.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:03.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:03.715+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:03.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:03.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:03.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:03.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:03.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:03.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:03.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:03.778+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:04.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:04.052+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38338 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:03.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:03.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:03.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:03.937+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:04.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:04.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:04.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:04.826+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:04.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:04.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:04.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:04.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:04.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:04.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:04.692+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:05.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:04.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:05.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:04.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:04.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:04.974+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:05.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:05.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:05.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:05.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:05.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:05.645+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:05.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:05.645+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:05.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:05.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:05.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:05.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:05.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:05.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:05.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:05.822+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:06.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:05.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:06.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:05.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:06.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:05.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:06.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:05.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:06.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:06.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:06.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:06.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:06.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:06.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:06.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:06.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:06.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:06.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:06.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:06.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:06.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:06.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:06.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:06.797+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:06.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:06.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:07.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:06.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:07.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:06.912+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:07.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:07.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:07.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:07.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:07.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:07.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:07.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:07.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:07.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:07.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:07.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:07.839+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:07.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:07.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:07.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:08.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:07.922+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:08.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:08.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:08.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:08.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:08.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:08.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:08.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:08.601+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:09.185 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:09.056+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38428 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:09.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:08.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:09.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:08.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:09.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:08.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:09.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:08.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:09.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:08.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:09.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:08.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:09.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:08.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:09.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:08.872+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:09.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:09.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:09.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:09.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:09.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:09.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:09.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:09.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:09.903 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:09.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:09.903 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:09.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:09.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:09.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:09.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:09.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:09.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:10.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:09.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:10.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:09.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:10.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:09.899+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:10.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:10.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:10.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:10.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:10.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:10.679+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:10.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:10.680+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:10.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:10.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:10.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:10.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:10.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:10.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:10.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:10.887+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:10.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:10.944+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:10.944+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:11.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:10.944+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:11.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:10.945+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:11.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:11.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:11.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:11.682+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:11.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:11.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:11.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:11.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:11.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:11.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:11.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:11.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:11.970+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:12.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:12.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:12.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:12.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:12.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:12.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:12.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:12.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:12.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:12.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:12.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:12.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:12.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:12.965+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:12.965+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:12.965+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:12.966+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:13.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:13.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:13.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:13.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:13.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:13.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:13.857+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:13.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:13.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:13.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:13.648+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:14.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:14.062+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38519 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:13.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:13.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:14.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:13.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:14.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:13.968+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:14.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:14.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:14.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:14.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:14.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:14.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:14.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:14.688+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:14.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:14.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:14.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:14.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:14.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:14.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:14.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:14.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:14.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:14.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:14.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:14.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:15.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:15.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:15.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:15.715+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:15.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:15.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:15.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:15.782+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:15.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:15.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:15.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:15.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:15.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:15.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:15.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:15.916+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:16.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:16.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:16.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:16.722+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:16.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:16.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:16.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:16.800+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:17.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:16.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:17.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:16.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:17.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:16.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:17.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:16.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:17.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:17.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:17.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:17.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:17.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:17.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:17.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:17.711+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:17.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:17.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:17.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:17.825+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:18.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:17.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:17.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:17.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:17.897+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:19.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:18.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:19.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:18.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:19.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:18.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:19.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:18.869+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:19.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:18.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:19.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:18.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:19.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:18.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:19.071 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:18.731+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:19.072 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:18.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:19.072 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:18.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:19.072 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:18.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:19.072 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:18.943+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:19.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:19.066+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38609 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:19.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:19.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:19.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:19.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:19.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:19.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:19.685+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:19.686+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:19.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:19.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:19.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:19.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:19.925+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:20.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:20.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:20.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:20.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:20.643+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:20.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:20.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:20.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:20.937+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:20.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:20.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:20.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:20.948+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:21.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:21.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:21.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:21.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:21.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:21.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:21.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:21.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:21.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:21.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:21.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:21.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:21.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:21.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:21.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:21.938+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:22.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:22.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:22.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:22.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:22.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:22.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:22.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:22.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:22.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:22.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:22.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:22.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:22.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:22.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:22.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:22.970+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:23.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:23.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:23.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:23.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:23.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:23.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:23.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:23.559+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:24.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:24.071+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38699 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:24.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:23.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:24.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:23.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:24.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:23.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:24.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:23.998+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:24.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:23.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:24.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:23.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:24.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:23.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:24.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:23.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:24.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:24.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:24.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:24.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:24.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:24.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:24.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:24.537+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:24.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:24.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:24.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:24.963+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:24.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:24.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:24.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:24.957+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:25.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:25.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:25.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:25.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:25.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:25.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:25.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:25.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:25.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:25.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:25.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:25.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:25.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:25.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:25.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:25.988+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:26.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:26.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:26.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:26.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:26.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:26.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:26.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:26.537+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:26.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:26.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:26.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:26.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:26.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:26.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:26.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:26.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:27.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:27.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:27.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:27.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:27.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:27.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:27.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:27.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:27.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:27.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:27.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:28.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:27.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:27.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:27.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:27.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:27.986+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:28.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:28.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:28.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:28.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:28.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:28.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:28.950 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:28.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:29.205 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:29.076+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38788 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:29.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:28.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:29.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:28.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:29.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:28.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:29.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:28.999+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:29.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:28.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:29.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:28.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:29.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:28.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:29.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:28.948+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:29.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:29.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:29.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:29.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:29.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:29.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:29.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:29.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:30.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:29.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:30.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:29.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:30.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:29.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:30.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:29.962+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:30.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:30.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:30.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:30.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:30.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:30.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:30.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:30.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:30.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:30.628+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:30.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:30.629+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:31.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:30.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:31.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:30.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:30.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:30.953+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:31.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:32.005 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:31.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:32.006 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:31.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:32.006 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:31.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:32.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:31.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:32.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:31.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:32.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:31.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:32.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:31.673+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:32.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:31.986+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:32.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:32.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:32.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:32.718+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:32.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:32.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:32.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:32.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:33.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:33.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:33.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:33.018+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:33.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:33.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:33.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:33.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:33.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:33.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:33.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:33.873+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:34.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:34.081+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38880 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:34.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:34.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:34.040+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:34.041+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:35.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:34.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:35.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:34.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:35.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:34.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:35.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:34.841+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:35.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:34.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:35.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:34.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:35.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:34.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:35.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:34.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:35.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:35.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:35.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:35.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:35.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:35.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:35.361 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:35.047+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:35.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:35.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:35.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:35.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:35.821+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:35.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:35.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:35.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:35.713+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.005+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:36.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:36.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:36.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:36.734+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:36.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:36.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:36.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:37.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:36.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:37.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:36.810+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:37.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:37.761+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:37.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:37.761+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:37.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:37.761+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:37.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:37.762+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:37.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:37.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:37.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:37.958+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:37.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:37.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:37.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:37.773+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:39.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:38.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:39.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:38.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:39.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:38.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:39.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:38.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:39.090 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:38.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:39.090 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:38.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:39.090 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:38.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:39.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:38.763+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:39.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:38.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:39.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:38.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:39.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:38.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:39.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:38.729+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:39.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:39.086+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 38969 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:39.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:39.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:39.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:39.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:39.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:39.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:39.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:39.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:39.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:39.810+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:39.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:39.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:39.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:39.902+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:40.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:40.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:40.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:40.816+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:40.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:40.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:40.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:40.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:40.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:40.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:40.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:41.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:40.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:41.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:41.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:41.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:41.883+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:41.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:41.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:42.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:41.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:42.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:41.777+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:41.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:41.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:41.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:41.768+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:42.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:42.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:42.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:42.815+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:42.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:42.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:43.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:42.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:43.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:42.788+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:42.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:42.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:42.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:42.843+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:43.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:43.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:43.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:43.822+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:43.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:43.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:43.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:43.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:43.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:43.765+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:43.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:43.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:43.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:43.851+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:44.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:44.091+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39060 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:44.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:44.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:44.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:44.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:44.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:44.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:44.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:44.792+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:44.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:44.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:44.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:44.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:44.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:44.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:44.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:44.786+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:44.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:44.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:44.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:44.898+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:45.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:45.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:45.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:45.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:45.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:45.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:45.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:46.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:45.740+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:45.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:45.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:45.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:45.822+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:46.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:46.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:46.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:46.846+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:46.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:46.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:46.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:47.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:46.777+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:47.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:46.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:47.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:46.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:47.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:46.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:47.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:46.845+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:47.865 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:47.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:47.865 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:47.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:47.865 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:47.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:47.865 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:47.734+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:47.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:47.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:47.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:47.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:47.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:47.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:47.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:48.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:47.861+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:48.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:47.862+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:49.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:48.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:49.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:48.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:49.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:48.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:49.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:48.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:49.100 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:48.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:49.100 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:48.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:49.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:48.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:49.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:48.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:49.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:48.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:49.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:48.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:49.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:48.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:49.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:48.912+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:49.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:49.096+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39149 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:49.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:49.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:49.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:49.899+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:49.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:49.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:49.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:49.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:49.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:49.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:49.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:49.774+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:50.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:50.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:50.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:50.938+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:50.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:50.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:50.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:50.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:50.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:50.800+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:50.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:50.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:50.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:50.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:50.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:50.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:50.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:50.878+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:51.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:51.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:51.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:51.971+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:51.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:51.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:51.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:51.772+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:52.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:51.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:52.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:51.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:52.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:51.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:52.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:51.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:52.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:52.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:52.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:52.998+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:52.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:52.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:52.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:52.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:52.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:52.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:52.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:52.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:53.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:53.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:53.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:53.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:53.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:53.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:53.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:53.784+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:53.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:53.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:53.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:53.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:53.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:53.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:53.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:53.887+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:54.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:54.100+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39241 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:53.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:53.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:53.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:53.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:54.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:54.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:54.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:54.831+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:54.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:55.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:54.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:55.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:54.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:55.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:54.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:54.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:54.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:54.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:55.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:54.945+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:55.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:55.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:55.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:55.942+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:55.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:55.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:56.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:55.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:56.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:55.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:55.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:55.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:55.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:55.872+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:56.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:56.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:56.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:56.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:56.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:56.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:56.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:56.751+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:56.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:56.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:56.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:57.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:56.845+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:58.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:57.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:58.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:57.766+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:58.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:57.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:58.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:57.768+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:58.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:57.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:58.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:57.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:58.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:57.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:58.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:57.871+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:58.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:58.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:58.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:58.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:59.066 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:58.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:59.066 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:58.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:59.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:58.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:39:59.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:58.840+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:59.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:58.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:59.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:58.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:59.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:58.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:39:59.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:58.791+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:39:59.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:59.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:59.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:59.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:59.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:59.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:39:59.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:39:59.105+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39330 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:39:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:39:59.064+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:00.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:59.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:00.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:59.744+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:00.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:59.744+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:00.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:39:59.745+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:00.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:59.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:00.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:59.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:00.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:59.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:00.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:39:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:39:59.877+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:00.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:00.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:00.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:00.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:00.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:00.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:00.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:00.715+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:00.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:00.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:00.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:00.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:00.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:00.920+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:01.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:01.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:01.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:01.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:01.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:01.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:01.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:01.681+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:01.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:01.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:01.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:01.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:01.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:01.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:01.913+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:02.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:02.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:02.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:02.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:03.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:02.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:03.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:02.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:03.009 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:02.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:03.009 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:02.726+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:03.009 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:02.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:03.009 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:02.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:03.009 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:02.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:03.010 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:02.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:03.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:03.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:03.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:03.069+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:03.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:03.744+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:03.922 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:03.744+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:03.922 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:03.744+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:03.922 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:03.745+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:04.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:04.110+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39421 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:03.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:03.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:04.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:03.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:04.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:03.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:04.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:04.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:04.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:04.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:04.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:04.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:04.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:04.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:05.079 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:04.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:05.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:04.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:05.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:04.721+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:05.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:04.722+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:05.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:04.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:05.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:04.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:05.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:04.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:05.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:04.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:05.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:05.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:05.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:05.077+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:05.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:05.730+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:05.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:05.730+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:05.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:05.730+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:06.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:05.731+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:05.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:05.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:05.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:05.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:06.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:06.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:06.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:06.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:07.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:06.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:07.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:06.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:07.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:06.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:07.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:06.729+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:07.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:06.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:07.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:06.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:07.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:06.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:07.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:06.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:07.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:07.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:07.141+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:07.141+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:07.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:07.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:07.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:07.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:07.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:07.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:07.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:07.741+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:07.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:07.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:07.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:07.922+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:08.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:08.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:08.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:08.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:09.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:08.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:09.095 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:08.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:09.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:08.741+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:09.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:08.742+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:09.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:08.968+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:09.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:08.968+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:09.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:08.968+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:09.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:08.969+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:09.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:09.115+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39509 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:09.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:09.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:09.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:09.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:09.093+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:10.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:09.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:10.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:09.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:10.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:09.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:10.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:09.990+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:10.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:09.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:10.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:09.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:10.131 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:09.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:10.131 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:09.777+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:10.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:10.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:10.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:10.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:11.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:10.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:11.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:10.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:11.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:10.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:11.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:10.797+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:11.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:11.085+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:11.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:11.085+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:11.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:11.085+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:11.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:11.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:11.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:11.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.007+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:11.086+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:12.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:11.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:12.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:11.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:12.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:11.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:12.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:11.819+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:12.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:12.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:12.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:12.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:11.988+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:12.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:12.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:12.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:12.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:13.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:12.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:13.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:12.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:13.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:12.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:13.032 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:12.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:13.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:13.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:13.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:13.029+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:13.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:13.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:13.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:13.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:13.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:13.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:13.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:13.843+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:14.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:14.120+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39594 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:14.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:14.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:14.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:14.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:14.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:14.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:14.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:14.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:15.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:14.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:15.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:14.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:15.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:14.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:15.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:14.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:15.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:15.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:15.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:15.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:15.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:15.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:15.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:15.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:15.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:15.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:15.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:15.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:15.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:15.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:15.838+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:15.839+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:16.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:16.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:16.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:16.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:16.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:16.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:16.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:16.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:16.224+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:16.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:16.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:16.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:16.879+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:17.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:17.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:17.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:17.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:17.052+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:17.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:17.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:17.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:17.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:18.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:18.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:18.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:18.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:17.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:17.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:17.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:17.848+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:18.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:18.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:18.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:18.244+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:19.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:18.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:19.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:18.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:19.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:18.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:19.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:18.861+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:19.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:19.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:19.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:19.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:19.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:19.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:19.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:19.055+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:19.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:19.125+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39682 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:19.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:19.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:19.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:19.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:19.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:19.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:19.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:19.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:20.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:20.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:20.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:20.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:20.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:20.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:20.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:20.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:20.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:20.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:20.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:20.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:20.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:20.873+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:21.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:21.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:21.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:21.055+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:21.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:21.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:21.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:21.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:21.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.013+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:21.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:21.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:21.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:21.835+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:22.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:22.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:22.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:22.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:22.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:22.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:22.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:22.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:23.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:22.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:23.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:22.874+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:23.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:23.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:23.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:23.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:23.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:23.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:23.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:23.224+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:24.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:23.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:24.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:23.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:24.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:23.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:24.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:23.918+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:24.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:23.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:24.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:23.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:24.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:23.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:24.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:23.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:24.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:24.130+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39773 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:24.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:24.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:24.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:24.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:24.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:24.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:24.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:24.936+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:24.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:24.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:24.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:24.974+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:25.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:25.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:25.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:25.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:25.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:25.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:25.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:25.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:25.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:25.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:25.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:25.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:26.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:26.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:26.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:26.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:26.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:26.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:26.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:26.857+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:26.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:26.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:26.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:26.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:27.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:27.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:27.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:27.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:27.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:27.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:27.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:27.831+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:27.955+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:27.955+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:27.955+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:27.956+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:28.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:28.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:28.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:28.345+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:29.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:28.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:29.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:28.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:29.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:28.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:29.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:28.834+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:29.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:28.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:29.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:28.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:29.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:28.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:29.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:28.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:29.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:29.135+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39862 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:29.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:29.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:29.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:29.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:30.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:29.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:29.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:29.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:29.843+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:29.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:29.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:29.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:29.959+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:30.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:30.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:30.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:30.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:30.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:30.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:30.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:30.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:30.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:30.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:30.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:30.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:30.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:30.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:30.961 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:30.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:30.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:30.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:30.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:30.958+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:31.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:31.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:31.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:31.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:31.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:31.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:31.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:31.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:31.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:31.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:32.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:31.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:32.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:31.841+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:32.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:32.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:32.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:32.356+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:32.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:32.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:32.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:32.931+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:32.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:32.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:33.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:32.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:33.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:32.832+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:33.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:33.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:33.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:33.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:33.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:33.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:33.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:33.307+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:33.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:33.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:33.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:33.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:33.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:33.817+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:33.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:33.818+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:34.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:34.140+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 39954 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:33.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:33.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:33.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:33.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:34.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:34.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:34.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:34.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:34.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:34.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:34.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:34.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:34.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:34.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:35.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:34.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:35.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:34.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:35.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:35.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:35.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:35.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:35.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:35.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:35.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:35.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:35.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:35.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:35.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:36.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:35.845+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:36.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:36.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:36.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:36.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:36.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:36.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:36.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:36.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:36.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:36.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:36.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:37.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:36.867+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:37.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:37.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:37.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:37.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:37.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:37.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:37.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:37.939+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:37.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:37.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:37.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:37.845+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:38.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:38.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:38.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:38.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:39.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:38.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:39.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:38.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:39.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:38.872+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:39.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:38.873+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:39.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:38.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:39.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:38.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:39.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:38.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:39.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:38.931+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:39.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:39.145+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40043 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:39.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:39.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:39.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:39.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:40.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:39.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:40.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:39.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:40.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:39.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:40.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:39.882+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:40.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:39.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:40.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:39.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:40.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:39.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:40.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:39.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:40.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:40.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:40.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:40.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:41.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:40.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:41.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:40.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:41.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:40.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:41.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:40.906+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:41.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:40.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:41.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:40.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:41.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:40.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:41.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:40.872+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:41.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:41.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:41.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:41.209+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:41.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:41.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:41.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:41.956+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:41.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:41.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:42.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:41.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:42.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:41.857+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:42.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:42.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:42.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:42.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:42.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:42.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:42.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:42.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:42.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:42.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:42.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:42.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:43.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:43.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:43.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:43.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:43.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:43.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:43.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:43.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:43.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:43.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:43.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:43.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:44.257 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:44.150+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40134 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:44.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:43.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:44.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:43.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:44.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:43.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:44.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:43.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:44.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:44.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:44.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:44.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:44.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:44.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:44.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:44.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:44.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:44.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:44.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:44.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:44.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:44.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:44.874+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:45.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:45.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:45.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:45.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:45.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:46.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:46.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:46.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:46.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:46.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:46.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:46.079 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:46.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:46.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:45.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:46.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:45.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:46.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:45.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:46.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:45.825+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:46.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:46.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:46.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:46.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:46.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:46.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:46.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:46.832+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:47.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:47.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:47.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:47.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:47.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:47.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:47.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:47.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:47.349+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:48.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:48.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:48.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:48.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:47.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:48.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:47.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:48.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:47.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:48.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:47.792+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:48.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:48.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:48.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:48.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:49.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:49.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:49.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:49.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.042+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:49.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:48.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:49.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:48.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:49.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:48.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:49.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:48.815+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:49.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:49.155+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40223 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:49.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:49.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:49.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:49.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:49.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:49.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:49.855+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:49.856+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:49.995+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:50.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:50.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:50.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:50.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:50.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:50.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:50.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:50.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:50.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:50.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:50.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:50.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:50.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:50.987+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:51.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:51.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:51.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:51.279+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:52.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:51.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:52.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:51.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:52.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:51.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:52.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:51.971+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:52.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:51.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:52.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:51.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:52.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:51.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:52.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:51.917+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:52.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:52.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:52.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:52.246+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:52.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:52.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:52.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:52.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:52.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:52.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:52.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:52.927+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:53.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:53.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:53.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:53.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:53.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:53.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:53.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:53.252+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:54.185 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:54.159+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40315 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:54.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:53.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:54.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:53.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:54.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:53.929+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:54.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:53.930+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:54.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:53.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:54.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:53.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:54.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:53.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:54.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:53.962+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:54.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:54.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:54.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:54.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:54.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:54.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:54.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:54.960+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:54.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:54.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:54.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:54.930+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:55.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:55.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:55.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:55.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:55.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:55.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:55.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:55.965+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:55.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:55.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:55.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:55.955+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:56.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:56.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:56.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:56.224+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:57.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:57.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:57.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:57.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:57.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:57.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:57.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:57.015+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:57.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:56.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:57.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:56.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:57.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:56.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:57.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:56.977+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:57.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:57.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:57.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:57.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:57.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:57.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:57.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:57.227+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:58.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:58.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:58.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:58.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:58.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:58.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:58.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:58.001+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:58.266 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:58.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:58.266 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:58.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:58.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:58.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:58.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:58.013+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:58.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:58.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:58.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:58.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:59.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:59.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:59.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:59.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:59.287 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:59.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:40:59.287 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:40:59.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:40:59.287 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:59.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:59.287 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:59.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:59.287 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:59.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:40:59.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:40:59.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:59.288 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:40:59.165+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40404 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:40:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:59.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:59.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:59.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:40:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:40:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:40:59.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:00.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:00.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:00.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:00.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:00.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:00.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:00.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:00.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:00.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:00.048+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:00.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:00.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:00.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:00.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:01.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:01.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:01.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:01.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:01.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:01.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:01.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:01.066+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:01.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:01.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:01.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:01.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:01.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:01.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:01.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:01.368+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:02.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:02.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:02.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:02.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:01.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:02.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:02.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:02.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:02.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:02.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:02.097+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:02.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:02.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:02.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:02.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:02.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:02.413+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:03.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:02.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:03.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:02.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:03.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:02.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:03.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:02.976+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:03.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:03.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:03.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:03.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:03.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:03.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:03.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:03.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:03.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:03.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:03.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:03.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:04.297 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:04.171+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40495 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:04.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:04.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:04.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:04.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:04.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:04.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:04.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:04.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:04.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:04.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:04.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:04.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.008+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:04.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:04.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:04.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:04.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:05.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:05.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:05.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:05.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:04.993+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:05.369 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:05.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:05.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:05.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:05.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:05.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:05.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:05.109+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:05.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:05.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:05.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:05.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:05.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:05.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:05.366+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:06.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:06.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:06.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:06.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:06.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:06.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:06.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:06.003+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:06.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:06.137+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:06.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:06.137+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:06.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:06.137+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:06.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:06.138+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:06.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:06.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:06.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:06.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:07.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:07.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:07.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:07.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:07.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:07.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:07.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:07.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:07.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:07.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:07.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:07.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:07.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:07.099+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:07.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:07.100+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:07.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:07.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:07.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:07.357+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:08.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:08.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:08.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:08.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:08.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:08.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:08.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:08.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:08.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:08.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:08.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:08.399+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:09.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:09.176+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40584 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:09.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:09.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:09.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:09.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:08.971+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:09.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:09.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:09.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:09.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:09.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:09.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:09.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:09.089+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:09.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:09.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:09.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:09.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:09.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:09.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:09.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:09.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:09.953+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:10.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:10.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:10.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:10.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:10.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:10.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:10.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:10.079+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:10.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:10.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:10.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:10.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:10.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:10.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:10.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:10.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:10.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:10.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:10.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:10.951+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:11.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:11.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:11.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:11.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:11.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:11.104+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:11.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:11.105+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:11.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:11.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:11.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:11.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:11.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:11.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:11.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:11.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:12.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:11.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:12.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:12.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:12.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:12.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:12.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:12.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:12.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:12.090+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:12.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:12.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:12.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:12.454+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:13.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:13.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:13.136+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:13.137+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:12.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:12.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:12.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:12.940+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:13.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:13.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:13.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:13.426+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:14.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:13.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:14.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:13.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:14.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:13.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:14.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:13.916+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:14.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:14.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:14.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:14.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:14.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:14.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:14.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:14.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:14.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:14.181+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40676 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:14.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:14.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:14.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:14.390+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:15.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:15.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:15.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:15.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:14.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:14.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:14.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:14.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:15.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:15.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:15.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:15.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:15.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:15.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:15.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:15.951+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:16.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:16.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:16.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:16.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:16.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:16.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:16.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:16.392+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:17.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:16.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:17.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:16.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:17.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:16.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:17.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:16.995+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:17.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:17.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:17.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:17.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:17.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:17.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:17.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:17.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:17.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:17.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:17.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:17.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:17.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:18.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:18.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:18.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:18.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:18.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:18.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.030+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:18.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:18.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:18.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:18.363+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:19.313 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:19.186+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40765 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:18.982+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:19.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:19.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:19.314 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:19.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:19.315 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:19.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:19.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:19.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:19.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:19.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:19.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:19.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:20.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:20.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:20.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:20.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:20.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:20.018+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:20.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:20.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:20.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:20.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:20.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:20.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:20.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:20.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:20.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:20.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:20.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:20.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:21.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:21.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:21.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:21.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:21.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:21.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:21.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:21.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:21.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:21.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:21.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:21.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:21.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:21.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:21.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:21.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:21.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:21.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:21.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:21.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:21.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:21.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:21.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:21.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:22.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:22.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:22.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:22.308+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:22.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:22.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:22.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:22.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:22.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:22.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:22.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:22.094+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:23.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:23.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:23.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:23.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:23.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:23.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:23.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:23.094+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:23.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:23.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:23.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:23.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:23.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:23.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:23.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:23.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:23.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:23.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:24.387 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:24.191+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40856 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:24.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:24.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:24.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:24.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:24.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:24.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:24.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:24.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:24.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:24.118+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:24.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:24.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:24.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:24.385+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:25.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:25.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:25.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:25.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:25.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:25.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:25.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:25.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:25.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:25.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:25.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:25.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:25.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:25.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:25.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:25.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:25.434+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:26.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:26.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:26.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:26.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:26.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:26.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:26.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:26.080+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:26.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:26.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:26.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:26.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:27.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:27.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:27.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:27.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:27.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:27.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:27.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:27.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:27.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:27.031+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:27.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:27.031+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:27.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:27.031+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:27.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:27.032+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:27.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:27.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:27.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:27.444+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:28.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:28.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:28.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:28.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:28.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:28.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:28.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:28.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:28.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:28.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:28.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:28.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:28.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:28.033+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:28.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:28.034+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:28.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:28.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:28.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:28.430+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:29.324 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:29.196+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 40945 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:29.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:29.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:29.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:29.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:29.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:29.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:29.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:29.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:29.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:29.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:29.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:29.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:29.326 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:29.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:29.326 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:29.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:29.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:29.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:29.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:29.443+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:30.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:30.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:30.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:30.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:30.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:30.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:30.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:30.154+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:30.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:30.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:30.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:30.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:30.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:30.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:30.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:30.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:30.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:30.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:30.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:30.435+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:31.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:31.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:31.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:31.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:31.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:31.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:31.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:31.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:31.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:31.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:31.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:31.471+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:32.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:32.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:32.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:32.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:31.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:32.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:32.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:32.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:32.140+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:32.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:32.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:32.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:32.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:32.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:32.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:32.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:32.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:33.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:33.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:33.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:33.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:33.176+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:32.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:32.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:32.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:33.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:32.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:33.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:33.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:33.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:33.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:34.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:34.201+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41037 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:34.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:34.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:34.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:34.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:34.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:34.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:30:58.156850+0000 (oldest deadline 2024-02-20T08:31:18.156850+0000) 2024-02-20T08:41:34.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:34.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:34.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:34.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:34.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:34.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:34.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:34.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:34.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:34.008+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:34.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:34.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:34.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:34.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:35.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:35.193+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:35.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:35.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:35.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:35.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:35.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:35.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:35.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:35.016+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:35.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:35.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:35.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:35.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:36.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:36.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:36.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:36.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:36.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:36.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:36.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:36.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:36.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:36.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:36.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:36.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:36.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:36.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:37.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:37.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:37.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:37.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:37.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:38.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:00.803179+0000 (oldest deadline 2024-02-20T08:31:20.803179+0000) 2024-02-20T08:41:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:37.989+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:38.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:38.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:38.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:31:07.255010+0000 (oldest deadline 2024-02-20T08:31:27.255010+0000) 2024-02-20T08:41:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:38.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:39.332 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:39.206+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41126 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:39.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:39.178+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:39.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:39.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:39.483+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:40.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:40.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:40.248 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:39.973+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:40.473+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:41.312 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:41.155+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:41.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:41.007+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:41.493+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:42.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:42.014+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:42.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:42.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:42.444+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:43.170+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:42.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:43.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:44.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:43.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:44.341 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:44.211+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41217 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:44.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:44.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:44.531+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:45.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:45.036+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:45.537+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:46.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:46.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:46.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:47.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:47.034+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:47.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:48.053+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:48.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:48.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:49.345 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:49.216+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41307 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:49.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:49.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:49.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:49.050+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:49.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:50.047+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:50.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:50.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:51.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:51.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:51.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:52.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:52.217+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:52.661+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:53.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:53.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:53.673+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:54.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:54.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:54.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:54.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:54.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:54.221+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41398 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:54.653+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:55.232+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:55.063+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:55.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:56.184+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:56.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:56.666+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:57.212+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:57.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:57.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:58.132+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:58.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:41:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:58.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:41:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:58.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:41:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:58.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:58.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:41:59.153+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:59.473 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:41:59.227+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41487 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:41:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:59.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:41:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:59.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:41:59.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:59.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:41:59.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:41:59.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:41:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:41:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:41:59.630+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:00.168+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:00.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:00.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:00.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:00.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:00.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:00.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:00.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:00.646+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:01.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:01.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:01.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:01.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:01.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:01.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:01.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:01.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:01.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:01.641+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:02.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:02.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:02.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:02.277+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:02.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:02.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:02.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:02.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:02.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:02.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:02.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:02.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:03.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:03.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:03.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:03.320+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:03.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:03.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:03.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:03.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:03.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:03.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:03.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:03.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:04.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:04.232+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41578 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:04.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:04.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:04.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:04.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:04.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:04.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:04.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:04.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:04.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:04.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:04.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:04.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:04.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:04.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:04.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:04.679+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:05.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:05.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:05.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:05.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:05.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:05.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:05.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:05.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:05.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:05.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:05.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:05.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:06.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:06.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:06.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:06.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:06.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:06.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:06.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:06.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:06.217+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:06.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:06.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:06.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:06.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:07.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:07.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:07.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:07.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:07.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:07.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:07.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:07.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:07.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:07.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:07.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:07.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:07.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:07.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:07.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:07.733+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:08.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:08.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:08.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:08.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:08.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:08.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:08.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:08.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:08.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:08.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:08.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:08.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:09.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:09.237+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41668 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:09.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:09.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:09.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:09.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:09.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:09.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:09.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:09.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:09.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:09.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:09.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:09.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:09.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:10.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:09.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:10.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:09.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:10.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:09.765+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:10.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:10.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:10.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:10.249+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:10.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:10.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:10.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:10.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:10.334+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:10.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:11.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:10.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:11.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:10.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:11.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:10.726+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:11.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:11.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:11.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:11.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:11.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:11.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:11.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:11.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:11.336+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:12.032 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:11.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:12.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:11.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:12.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:11.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:12.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:11.764+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:12.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:12.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:12.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:12.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:12.283+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:12.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:12.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:12.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:12.728+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:13.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:13.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:13.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:13.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:13.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:13.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:13.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:13.315+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:13.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:13.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:13.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:13.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:13.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:13.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:13.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:13.295+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:13.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:13.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:13.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:13.775+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:14.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:14.242+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41759 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:14.251+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:14.251+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:14.251+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:14.252+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:14.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:14.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:14.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:14.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:14.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:15.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:14.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:15.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:14.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:15.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:14.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:15.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:14.809+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:15.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:15.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:15.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:15.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:15.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:15.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:15.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:15.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:16.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:15.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:16.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:15.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:16.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:15.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:16.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:15.841+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:16.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:16.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:16.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:16.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:16.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:16.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:16.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:16.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:17.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:16.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:17.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:16.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:17.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:16.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:17.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:16.825+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:17.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:17.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:17.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:17.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:17.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:17.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:17.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:17.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:18.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:17.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:18.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:17.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:18.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:17.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:18.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:17.782+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:18.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:18.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:18.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:18.138+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:18.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:18.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:18.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:18.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:19.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:18.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:19.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:18.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:19.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:18.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:19.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:18.746+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:19.377 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:19.248+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41848 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:19.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:19.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:19.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:19.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:19.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:19.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:19.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:19.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:19.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:19.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:20.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:19.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:20.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:19.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:20.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:19.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:20.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:19.751+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:20.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:20.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:20.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:20.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:20.096+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:21.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:20.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:21.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:20.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:21.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:20.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:21.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:20.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:21.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:21.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:21.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:21.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:21.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:21.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:21.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:21.124+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:22.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:21.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:22.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:21.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:22.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:21.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:22.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:21.785+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:22.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:22.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:22.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:22.130+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:22.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:22.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:22.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:22.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:22.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:22.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:23.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:22.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:23.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:22.825+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:23.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:23.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:23.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:23.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:23.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:23.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:23.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:23.097+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:23.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:23.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:23.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:23.858+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:24.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:24.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:24.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:24.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:24.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:24.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:24.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:24.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:24.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:24.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:24.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:24.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:24.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:24.129+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:24.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:24.130+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:24.385 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:24.253+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 41939 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:24.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:24.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:24.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:24.832+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:25.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:25.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:25.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:25.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:25.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:25.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:25.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:25.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:25.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:25.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:25.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:25.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:25.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:25.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:25.839+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:26.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:26.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:26.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:26.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:26.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:26.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:26.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:26.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:26.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:26.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:26.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:26.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:26.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:26.810+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:27.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:27.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:27.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:27.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:27.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:27.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:27.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:27.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:27.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:27.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:27.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:27.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:27.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:27.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:27.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:27.243+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:27.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:28.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:27.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:28.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:27.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:28.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:27.797+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:28.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:28.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:28.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:28.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:28.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:28.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:28.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:28.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:28.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:29.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:28.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:29.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:28.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:29.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:28.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:29.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:28.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:29.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:29.258+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42029 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:29.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:29.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:29.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:29.282+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:29.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:29.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:29.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:29.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:30.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:29.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:30.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:29.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:30.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:29.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:30.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:29.853+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:30.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:30.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:30.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:30.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:30.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:30.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:30.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:30.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:30.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:30.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:30.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:30.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:30.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:30.168+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:30.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:30.169+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:31.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:30.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:31.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:30.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:31.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:30.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:31.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:30.808+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:31.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:31.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:31.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:31.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:31.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:31.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:31.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:31.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:31.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:31.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:31.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:31.297+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:31.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:31.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:31.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:31.853+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:32.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:32.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:32.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:32.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:32.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:32.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:32.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:32.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:32.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:32.233+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:32.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:32.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:32.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:32.822+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:33.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:33.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:33.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:33.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:33.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:33.273+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:33.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:33.274+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:33.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:33.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:33.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:33.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:33.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:33.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:33.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:33.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:33.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:34.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:33.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:34.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:33.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:34.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:33.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:34.723 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:34.264+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42120 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:34.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:34.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:34.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:34.293+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:34.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:34.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:34.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:34.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:34.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:35.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:34.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:35.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:34.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:35.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:34.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:35.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:34.851+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:35.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:35.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:35.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:35.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:35.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:35.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:35.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:35.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:35.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:35.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:35.194+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:36.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:35.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:36.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:35.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:36.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:35.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:36.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:35.888+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:36.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:36.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:36.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:36.220+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:36.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:36.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:36.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:36.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:37.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:36.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:37.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:36.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:37.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:36.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:37.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:36.904+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:37.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:37.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:37.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:37.212+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:37.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:37.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:37.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:37.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:38.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:38.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:38.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:38.166+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:38.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:38.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:38.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:38.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:38.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:37.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:38.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:37.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:38.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:37.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:38.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:37.930+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:39.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:38.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:39.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:38.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:39.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:38.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:39.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:38.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:39.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:39.270+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42209 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:39.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:39.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:39.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:39.163+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:39.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:39.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:39.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:39.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:39.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:39.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:39.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:39.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:40.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:40.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:40.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:40.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:40.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:40.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:40.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:40.139+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:40.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:40.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:40.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:40.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:41.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:41.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:41.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:41.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:41.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:41.181+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:41.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:41.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:41.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:41.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:41.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:42.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:41.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:42.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:41.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:42.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:41.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:42.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:41.943+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:42.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:42.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:42.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:42.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:42.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:42.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:42.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:42.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:43.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:42.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:43.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:42.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:43.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:42.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:43.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:42.959+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:43.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:43.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:43.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:43.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:43.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:43.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:43.190+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:43.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:43.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:43.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:43.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:43.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:43.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:43.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:43.922+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:44.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:44.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:44.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:44.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:44.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:44.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:44.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:44.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:44.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:44.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:44.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:44.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:44.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:44.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:44.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:44.143+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:44.402 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:44.275+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42300 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:45.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:45.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:45.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:45.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:44.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:44.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:45.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:44.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:45.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:44.955+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:45.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:45.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:45.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:45.256+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:46.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:45.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:46.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:45.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:46.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:45.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:46.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:45.987+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:46.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:46.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:46.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:46.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:46.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:46.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:46.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:46.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:47.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:47.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:47.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:47.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:47.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:47.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:47.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:47.130+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:47.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:47.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:47.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:47.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:47.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:47.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:47.032+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:47.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:47.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:47.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:47.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:48.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:48.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:48.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:48.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:48.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:48.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:48.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:48.153+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:48.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:48.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:48.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:48.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:48.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:48.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:48.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:48.041+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:48.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:48.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:48.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:48.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:49.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:49.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:49.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:49.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:49.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:49.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:49.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:49.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:49.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:49.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:49.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:49.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:49.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:49.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:49.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:49.002+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:49.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:49.281+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42390 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:49.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:49.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:49.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:49.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:50.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:50.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:50.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:50.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:50.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:50.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:50.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:50.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:50.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:50.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:50.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:50.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:50.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:50.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:50.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:50.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:50.411+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:51.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:51.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:51.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:51.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:51.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:51.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:51.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:51.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:51.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:51.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:51.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:51.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:50.986+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:51.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:51.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:51.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:51.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:51.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:52.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:52.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:52.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:52.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:52.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:52.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:52.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:52.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:52.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:52.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:52.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:52.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:52.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:52.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:52.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:52.191+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:53.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:53.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:53.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:53.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:53.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:53.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:53.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:53.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:53.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:53.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:53.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:53.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:53.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:53.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:53.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:53.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:53.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:53.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:53.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:53.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:53.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:53.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:53.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:53.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:54.356 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:54.286+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42481 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:54.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:54.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:54.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:54.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:54.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:54.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:54.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:54.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:54.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:54.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:54.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:54.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:54.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:54.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:54.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:54.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:55.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:55.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:55.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:55.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:55.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:55.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:55.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:55.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:55.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:55.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:55.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:55.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.018+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:55.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:55.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:55.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:55.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:56.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:56.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:56.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:56.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:56.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:56.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:56.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:56.186+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:56.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:56.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:56.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:56.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:55.980+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:56.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:56.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:56.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:56.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:57.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:57.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:57.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:57.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:57.227 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:57.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:57.227 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:57.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:57.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:56.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:57.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:56.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:57.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:56.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:57.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:56.959+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:57.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:57.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:57.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:57.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:57.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:57.943+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:57.943+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:57.943+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:57.944+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:58.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:58.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:58.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:58.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:58.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:58.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:58.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:58.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:59.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:59.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:59.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:59.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:59.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:59.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:42:59.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:42:59.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:59.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:58.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:58.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:58.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:42:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:58.982+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:42:59.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:42:59.291+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42571 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:42:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:59.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:59.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:59.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:42:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:42:59.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:00.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:00.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:00.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:00.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:59.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:59.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:59.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:42:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:42:59.947+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:00.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:00.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:00.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:00.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:00.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:00.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:00.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:01.184 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:01.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:01.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:01.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:01.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:01.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:01.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:01.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:01.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:00.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:01.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:00.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:01.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:00.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:01.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:00.910+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:01.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:01.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:01.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:01.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:02.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:01.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:02.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:01.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:02.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:01.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:02.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:01.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:02.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:02.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:02.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:02.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:02.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:02.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:02.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:02.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:03.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:02.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:03.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:02.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:03.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:02.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:03.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:02.828+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:03.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:03.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:03.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:03.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:03.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:03.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:03.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:03.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:03.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:03.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:03.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:03.828+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:04.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:04.297+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42660 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:04.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:04.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:04.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:04.237+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:04.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:04.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:04.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:04.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:04.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:04.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:04.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:04.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:05.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:04.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:05.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:04.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:05.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:04.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:05.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:04.800+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:05.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:05.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:05.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:05.235+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:05.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:05.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:05.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:05.192+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:06.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:05.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:06.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:05.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:06.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:05.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:06.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:05.832+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:06.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:06.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:06.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:06.211+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:06.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:06.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:06.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:06.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:06.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:06.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:06.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:07.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:06.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:07.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:06.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:07.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:06.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:07.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:06.793+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:07.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:07.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:07.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:07.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:07.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:07.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:07.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:07.227+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:07.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:07.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:07.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:07.748+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:08.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:08.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:08.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:08.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:08.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:08.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:08.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:08.160+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:08.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:08.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:08.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:08.786+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:09.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:09.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:09.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:09.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:09.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:09.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:09.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:09.162+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:09.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:09.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:09.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:09.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:09.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:09.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:09.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:09.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:09.429 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:09.302+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42751 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:10.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:09.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:10.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:09.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:10.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:09.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:10.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:09.786+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:10.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:10.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:10.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:10.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:10.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:10.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:10.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:10.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:10.168+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:11.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:10.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:11.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:10.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:11.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:10.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:11.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:10.742+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:11.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:11.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:11.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:11.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:11.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:11.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:11.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:11.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:12.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:11.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:12.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:11.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:12.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:11.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:12.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:11.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:12.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:12.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:12.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:12.197+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:12.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:12.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:12.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:12.208+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:13.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:12.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:13.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:12.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:13.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:12.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:13.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:12.793+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:13.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:13.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:13.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:13.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:13.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:13.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:13.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:13.251+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:14.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:13.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:14.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:13.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:14.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:13.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:14.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:13.756+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:14.435 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:14.308+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42840 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:14.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:14.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:14.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:14.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:14.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:14.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:14.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:14.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:14.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:14.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:14.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:14.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:14.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:14.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:14.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:14.268+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:15.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:14.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:15.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:14.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:15.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:14.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:15.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:14.735+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:15.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:15.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:15.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:15.281+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:15.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:15.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:15.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:15.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:16.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:15.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:16.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:15.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:16.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:15.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:16.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:15.748+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:16.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:16.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:16.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:16.197+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:16.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:16.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:16.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:16.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:16.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:16.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:16.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:16.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:17.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:17.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:17.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:17.164+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:17.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:17.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:17.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:17.266+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:17.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:17.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:17.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:17.712+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:18.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:18.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:18.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:18.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:18.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:18.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:18.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:18.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:19.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:18.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:19.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:18.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:19.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:18.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:19.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:18.746+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:19.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:19.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:19.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:19.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:19.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:19.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:19.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:19.182+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:19.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:19.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:19.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:19.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:19.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:19.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:19.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:19.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:19.440 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:19.313+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 42930 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:20.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:19.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:20.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:19.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:20.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:19.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:20.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:19.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:20.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:20.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:20.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:20.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:20.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:20.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:20.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:20.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:21.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:20.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:21.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:20.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:21.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:20.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:21.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:20.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:21.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:21.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:21.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:21.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:21.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:21.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:21.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:21.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:22.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:21.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:22.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:21.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:22.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:21.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:22.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:21.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:22.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:22.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:22.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:22.208+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:22.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:22.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:22.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:22.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:22.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:22.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:22.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:22.717+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:23.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:23.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:23.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:23.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:23.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:23.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:23.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:23.241+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:23.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:23.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:23.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:23.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:23.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:23.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:23.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:23.250+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:23.738+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:23.738+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:23.738+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:23.738+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:24.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:24.319+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43020 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:24.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:24.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:24.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:24.212+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:24.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:24.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:24.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:24.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:24.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:24.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:24.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:24.708+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:25.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:25.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:25.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:25.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:25.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:25.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:25.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:25.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:25.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:25.208+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:25.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:25.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:25.678+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:25.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:26.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:26.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:26.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:26.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:26.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:26.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:26.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:26.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:26.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:26.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:26.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:26.646+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:27.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:27.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:27.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:27.208+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:27.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:27.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:27.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:27.264+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:27.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:27.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:27.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:27.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:27.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:27.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:27.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:28.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:28.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:28.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:28.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:28.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:28.217+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:28.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:28.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:28.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:28.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:28.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:28.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:28.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:28.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:28.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:28.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:28.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:29.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:29.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:29.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:29.224+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:29.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:29.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:29.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:29.204+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:29.474 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:29.324+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43110 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:29.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:29.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:29.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:29.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:29.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:29.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:29.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:30.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:30.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:30.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:30.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:30.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:30.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:30.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:30.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:30.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:30.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:30.192+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:30.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:30.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:30.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:30.627+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:31.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:31.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:31.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:31.223+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:31.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:31.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:31.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:31.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:31.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:31.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:31.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:31.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:31.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:31.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:31.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:31.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:32.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:32.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:32.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:32.180+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:32.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:32.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:32.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:32.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:32.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:32.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:32.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:32.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:33.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:33.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:33.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:33.200+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:33.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:33.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:33.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:33.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:33.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:33.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:33.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:33.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:34.452 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:34.329+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43199 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:34.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:34.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:34.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:34.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:34.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:34.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:34.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:34.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:34.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:34.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:34.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:34.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:34.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:34.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:34.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:34.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:34.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:34.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:34.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:34.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:35.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:35.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:35.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:35.125+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:35.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:35.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:35.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:35.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:35.195+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:35.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:35.641+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:35.641+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:35.641+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:35.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:35.642+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:36.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:36.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:36.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:36.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:36.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:36.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:36.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:36.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:36.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:36.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:36.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:36.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:37.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:37.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:37.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:37.181+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:37.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:37.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:37.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:37.125+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:37.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:37.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:37.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:37.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:38.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:38.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:38.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:38.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:38.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:38.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:38.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:38.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:38.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:38.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:38.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:38.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:38.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:38.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:38.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:38.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:38.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:38.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:38.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:38.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:38.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:39.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:39.335+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43278 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:39.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:39.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:39.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:39.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:39.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:39.186+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:39.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:39.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:39.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:39.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:39.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:39.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:39.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:39.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:39.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:39.102+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:39.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:39.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:39.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:39.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:40.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:40.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:40.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:40.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:40.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:40.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:40.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:40.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:40.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:40.085+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:40.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:40.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:40.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:40.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:41.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:41.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:41.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:41.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:41.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:41.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:41.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:41.047+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:41.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:41.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:41.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:41.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:41.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:41.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:41.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:41.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:42.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:42.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:42.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:42.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:42.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:42.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:42.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:42.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:42.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:42.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:42.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:42.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:42.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:42.096+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:42.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:42.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:42.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:42.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:42.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:43.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:43.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:43.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:43.055+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:43.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:43.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:43.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:43.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:43.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:43.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:43.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:43.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:43.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:43.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:43.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:43.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:44.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:44.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:44.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:44.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:44.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:44.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:44.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:44.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:44.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:44.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:44.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:44.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:44.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:44.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:44.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:44.040+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:44.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:44.341+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43358 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:45.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:44.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:45.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:44.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:45.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:44.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:45.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:44.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:45.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:45.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:45.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:45.104+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:45.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:45.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:45.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:45.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:45.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:45.044+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:46.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:45.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:46.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:45.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:46.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:45.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:46.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:45.742+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:46.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:46.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:46.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:46.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:46.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:46.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:46.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:46.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:46.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:46.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:47.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:46.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:47.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:46.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:47.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:46.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:47.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:46.753+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:47.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:47.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:47.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:47.034+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:47.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:47.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:47.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:47.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:47.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:47.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:47.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:47.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:47.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:47.748+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:48.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:48.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:48.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:48.015+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:48.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:48.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:48.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:48.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:48.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:48.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:48.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:49.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:48.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:49.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:48.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:49.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:48.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:49.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:48.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:49.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:49.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:49.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:49.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:49.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:49.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:49.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:49.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:49.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:49.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:49.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:49.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:49.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:49.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:49.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:49.095+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:49.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:49.347+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43439 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:50.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:49.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:50.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:49.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:50.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:49.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:50.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:49.774+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:50.061 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:50.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:50.061 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:50.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:50.061 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:50.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:50.061 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:50.031+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:50.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:50.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:50.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:50.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:50.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:50.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:50.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:50.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:51.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:50.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:51.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:50.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:51.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:50.759+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:51.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:50.760+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:51.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:51.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:51.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:51.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:51.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:51.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:51.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:51.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:51.032+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:52.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:51.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:52.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:51.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:52.005 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:51.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:52.005 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:51.728+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:52.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:52.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:52.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:52.002+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:52.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:52.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:52.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:52.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:53.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:52.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:53.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:52.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:53.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:52.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:53.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:52.756+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:53.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:53.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:53.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:53.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:53.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:53.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:53.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:53.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:53.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:53.062+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:53.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:53.062+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:53.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:53.062+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:53.410 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:53.063+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:53.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:53.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:53.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:53.751+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:54.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:54.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:54.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:54.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:54.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:54.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:54.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:54.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:54.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:54.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:54.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:54.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:54.084+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:54.353+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43520 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:43:55.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:54.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:55.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:54.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:55.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:54.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:55.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:54.799+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:55.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:55.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:55.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:55.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:55.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:55.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:55.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:55.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:55.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:56.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:56.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:56.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:56.021+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:56.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:56.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:56.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:56.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:56.030+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:55.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:55.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:55.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:55.809+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:57.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:57.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:57.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:57.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:57.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:57.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:56.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:56.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:56.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:56.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:58.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:58.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:58.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.990+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:58.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:57.992+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:58.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:57.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:58.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:57.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:58.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:57.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:58.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:57.782+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:58.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:58.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:58.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:58.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:59.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:58.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:59.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:58.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:59.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:58.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:43:59.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:58.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:59.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:58.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:59.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:58.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:59.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:58.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:43:59.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:58.797+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:59.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:59.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:59.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:59.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:59.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:59.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:43:59.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:43:59.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:43:59.374 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:43:59.359+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43603 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:00.110 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:59.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:00.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:59.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:00.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:59.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:00.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:43:59.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:00.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:59.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:00.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:59.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:00.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:59.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:00.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:43:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:43:59.795+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:00.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:00.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:00.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:00.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:01.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:01.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:01.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:01.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:01.084+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:00.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:00.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:00.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:00.963+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:01.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:00.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:01.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:00.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:01.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:00.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:01.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:00.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:02.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:01.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:02.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:01.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:02.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:01.797+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:02.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:01.797+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:02.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:01.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:02.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:01.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:02.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:01.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:02.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:01.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:02.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:02.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:02.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:02.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:02.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:02.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:02.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:02.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:02.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:02.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:02.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:02.821+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:02.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:02.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:02.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:02.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:03.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:03.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:03.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:03.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:03.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:03.084+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:03.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:04.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:03.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:04.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:03.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:04.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:03.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:04.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:03.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:04.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:03.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:04.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:03.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:04.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:03.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:04.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:04.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:04.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:04.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:04.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:04.082+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:04.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:04.365+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43693 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:05.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:05.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:05.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:05.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:05.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:04.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:05.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:04.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:05.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:04.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:05.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:04.920+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:04.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:04.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:04.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:04.866+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:05.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:05.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:05.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:05.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:05.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:05.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:05.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:05.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:05.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:05.853+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:06.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:06.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:06.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:06.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:07.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:07.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:07.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:07.070+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:07.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:06.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:07.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:06.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:07.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:06.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:07.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:06.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:07.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:06.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:07.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:06.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:07.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:06.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:07.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:06.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:08.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:08.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:08.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:08.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:08.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:07.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:08.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:07.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:08.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:07.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:08.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:07.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:08.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:07.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:08.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:07.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:08.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:07.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:08.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:07.889+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:09.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:09.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:09.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:09.068+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:09.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:08.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:09.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:08.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:09.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:08.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:09.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:08.902+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:09.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:08.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:09.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:08.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:09.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:08.921+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:09.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:08.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:09.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:09.370+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43784 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:10.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:10.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:10.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:10.104+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:10.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:09.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:10.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:09.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:10.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:09.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:10.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:09.920+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:10.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:09.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:10.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:09.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:10.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:09.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:10.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:09.916+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:11.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:11.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:11.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:11.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:11.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:10.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:11.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:10.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:11.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:10.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:11.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:10.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:11.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:10.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:11.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:10.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:11.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:10.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:11.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:10.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:12.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:11.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:12.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:11.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:12.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:11.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:12.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:11.967+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:12.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:11.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:12.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:11.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:12.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:11.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:12.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:11.892+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:12.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:13.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:12.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:13.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:12.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:13.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:12.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:13.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:12.892+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:13.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:12.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:13.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:12.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:13.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:12.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:13.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:12.962+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:13.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:13.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:13.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:13.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:13.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:13.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:13.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:13.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:13.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:13.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:13.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:13.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:13.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:13.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:14.400 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:14.376+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43874 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:14.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:14.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:14.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:14.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:14.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:14.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:14.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:14.144+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:15.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:15.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:15.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:15.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:15.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:14.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:15.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:14.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:15.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:14.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:15.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:14.965+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:14.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:14.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:14.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:15.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:14.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:16.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:16.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:16.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:16.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:15.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:16.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:15.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:16.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:15.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:16.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:15.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:16.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:15.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:16.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:15.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:16.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:15.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:16.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:15.954+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:17.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:17.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:17.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:17.132+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:17.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:16.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:17.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:16.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:17.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:16.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:17.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:16.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:17.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:16.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:17.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:16.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:17.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:16.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:17.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:16.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:18.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:18.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:18.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:18.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:18.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:18.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:18.138+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:18.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:17.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:18.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:17.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:18.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:17.964+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:18.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:17.965+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:18.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:17.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:18.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:17.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:18.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:17.985+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:18.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:17.986+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:19.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:19.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:19.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:19.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:19.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:19.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:19.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:19.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:19.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:18.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:19.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:18.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:19.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:18.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:19.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:18.980+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:19.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:19.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:19.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:19.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:19.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:19.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:19.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:19.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:19.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:19.380+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 43963 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:20.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:20.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:20.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:20.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:20.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:20.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:20.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:20.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:20.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:20.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:20.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:20.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:20.073+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:21.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:21.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:21.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:21.119+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:21.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:21.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:21.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:21.043+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:21.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:21.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:21.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:21.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:21.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:22.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:22.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:22.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:22.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:22.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:22.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:22.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:22.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:22.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:22.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:22.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:22.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:22.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:22.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:23.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:23.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:23.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:23.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.013+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:23.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:23.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:23.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:23.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:24.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:24.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:24.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:24.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:24.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:24.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:24.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:24.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:24.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:24.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:24.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:24.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:23.970+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:23.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:24.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:24.387+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44053 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:24.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:24.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:24.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:24.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:25.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:25.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:25.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:25.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:25.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:25.092+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:24.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:24.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:24.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:24.958+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:25.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:25.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:25.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:25.986+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:26.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:26.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:26.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:26.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:26.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:26.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:26.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:25.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:25.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:25.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:26.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:25.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:26.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:26.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:26.984+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:26.985+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:27.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:27.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:27.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:27.127+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:26.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:26.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:26.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:26.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:28.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:28.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:28.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:28.032+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:28.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:28.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:28.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:28.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:28.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:28.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:28.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:27.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:28.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:27.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:28.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:27.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:28.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:27.927+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:29.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:29.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:29.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:29.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:29.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:28.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:28.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:28.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:28.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:29.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:29.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:29.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:29.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:29.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:29.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:29.391+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44132 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:30.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:30.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:30.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:30.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:30.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:30.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:30.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:30.040+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:30.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:29.854+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:30.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:29.854+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:30.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:29.854+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:30.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:29.855+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:30.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:30.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:30.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:30.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:31.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:31.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:31.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:31.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:31.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:31.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:31.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:31.003+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:31.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:30.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:31.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:30.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:31.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:30.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:31.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:30.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:31.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:31.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:31.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:31.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:32.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:31.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:32.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:31.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:32.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:31.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:32.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:31.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:32.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:32.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:32.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:32.048+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:32.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:32.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:32.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:32.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:32.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:32.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:32.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:32.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:32.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:32.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:32.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:33.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:33.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:33.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:33.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:33.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:33.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:33.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:33.061+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:33.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:33.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:33.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:33.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:33.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:33.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:33.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:33.101+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:33.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:33.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:33.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:33.880+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:34.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:34.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:34.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:34.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:34.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:34.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:34.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:34.100+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:34.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:34.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:34.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:34.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:34.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:34.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:34.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:34.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:34.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:34.398+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44177 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:34.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:35.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:34.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:35.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:34.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:35.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:34.903+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:35.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:35.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:35.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:35.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:35.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:35.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:35.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:35.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:35.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:35.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:35.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:35.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:35.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:35.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:35.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:36.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:36.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:36.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:36.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:36.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:36.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:36.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:36.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:36.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:35.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:35.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:35.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:35.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:37.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:37.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:37.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:37.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:36.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:36.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:36.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:36.930+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:37.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:37.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:37.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:37.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:37.121+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:38.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:38.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:38.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:38.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:37.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:37.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:38.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:37.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:38.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:37.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:38.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:38.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:38.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:38.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:39.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:39.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:39.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:39.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:39.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:39.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:39.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:39.057+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:39.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:39.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:39.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:39.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:39.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:39.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:39.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:39.121+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:39.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:38.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:39.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:38.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:39.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:38.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:39.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:38.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:39.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:39.404+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44222 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:40.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:40.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:40.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:40.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:40.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:40.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:40.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:40.127+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:40.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:40.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:40.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:40.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:40.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:40.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:40.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:40.073+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:40.263 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:39.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:40.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:39.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:40.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:39.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:40.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:39.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:40.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:40.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:40.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:40.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:41.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:41.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:41.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:41.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:41.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:41.062+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:41.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:41.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:41.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:41.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:42.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:41.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:42.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:41.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:42.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:41.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:42.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:41.987+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:42.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:42.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:42.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:42.223+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:42.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:42.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:42.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:42.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:42.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:42.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:42.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:42.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:42.945+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:43.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:43.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:43.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:43.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:43.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:43.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:43.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:43.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:43.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:44.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:44.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:44.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:44.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:44.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:44.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:44.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:44.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:44.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:44.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:44.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:44.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:44.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:44.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:44.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:44.224+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:44.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:43.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:44.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:43.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:44.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:43.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:44.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:43.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:44.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:44.410+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44267 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:45.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:45.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:45.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:45.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:44.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:44.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:44.968+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:45.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:44.969+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:45.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:45.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:45.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:45.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:45.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:45.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:45.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:45.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:46.239 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:46.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:46.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:46.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:46.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:46.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:46.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:46.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:46.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:45.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:46.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:45.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:46.241 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:45.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:46.241 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:45.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:46.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:46.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:46.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:46.237+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:46.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:46.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:46.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:46.948+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:47.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:47.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:47.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:47.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:47.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:47.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:47.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:47.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:47.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:48.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:48.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:48.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:48.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:48.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:48.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:48.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:48.021+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:48.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:47.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:48.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:47.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:48.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:47.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:48.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:47.978+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:48.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:48.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:48.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:48.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:49.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:49.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:49.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:49.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:49.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:49.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:49.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:49.070+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:49.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:49.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:49.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:49.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:49.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:49.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:49.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:49.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:49.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:49.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:49.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:49.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.028+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:49.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:49.415+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44313 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:50.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:50.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:50.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:50.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:50.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:50.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:50.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:50.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:50.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:50.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:50.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:50.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:49.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:50.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:50.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:50.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:50.300+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:51.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:51.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:51.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:51.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:51.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:51.009+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:51.311 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:51.009+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:51.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:51.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:51.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:51.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:51.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:51.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:51.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:51.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:51.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:51.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:51.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:51.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:52.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:52.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:52.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:52.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:52.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:52.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:52.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:52.050+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:52.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:52.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:52.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:52.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:52.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:52.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:52.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:52.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:53.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:53.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:53.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:53.093+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:53.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:53.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:53.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:53.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:53.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:53.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:53.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:53.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:53.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:53.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:53.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:53.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:53.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:53.305+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:53.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:53.306+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:54.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:54.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:54.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:54.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:54.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:54.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:54.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:54.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:54.425 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:54.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:54.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:54.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:54.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:54.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:54.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:54.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:54.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:54.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:54.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:54.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:54.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:54.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:54.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:54.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:54.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:54.421+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44358 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:44:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:55.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:55.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:55.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:55.084+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:55.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:55.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:55.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:55.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:55.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:55.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:55.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:55.310+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:56.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:56.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:56.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:56.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:56.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:56.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:56.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:56.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:56.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:56.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:56.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:56.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:56.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:56.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:57.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:57.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:57.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:57.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:57.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:57.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:57.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:57.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:57.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:57.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:57.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:57.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:57.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:57.267+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:58.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:58.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:58.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:58.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:58.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:58.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:58.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:58.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:58.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:58.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:58.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:58.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:58.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:58.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:58.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:58.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:58.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:59.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:59.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:59.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:59.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:59.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:59.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:44:59.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:44:59.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:59.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:59.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:59.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:59.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:59.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:59.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:44:59.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:44:59.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:59.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:59.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:59.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:44:59.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:44:59.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:44:59.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:44:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:44:59.427+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44403 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:00.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:00.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:00.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:00.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:00.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:00.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:00.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:00.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:00.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:00.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:00.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:00.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:00.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:00.244+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:01.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:01.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:01.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:01.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:01.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:01.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:01.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:01.106+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:01.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:01.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:01.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:01.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:01.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:01.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:01.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:01.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:01.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:01.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:01.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:01.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:02.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:02.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:02.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:02.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:02.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:02.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:02.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:02.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:02.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:02.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:02.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:02.083+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:02.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:02.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:02.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:02.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:02.244+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:03.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:03.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:03.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:03.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:03.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:03.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:03.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:03.200+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:03.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:03.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:03.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:03.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:03.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:04.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:04.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:04.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:04.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:04.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:04.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:04.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:04.100+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:04.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:04.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:04.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:04.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:04.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:04.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:04.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:04.215+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:04.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:04.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:04.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:04.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:04.057+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:04.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:04.432+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44448 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:05.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:05.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:05.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:05.116+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:05.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:05.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:05.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:05.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:05.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:05.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:05.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:05.041+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:06.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:06.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:06.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:06.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:06.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:06.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:06.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:06.153+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:06.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:06.154+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:06.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:06.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:06.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:06.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:07.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:07.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:07.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:07.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:07.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:07.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:07.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:07.055+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:07.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:07.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:07.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:07.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:08.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:08.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:08.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:08.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:08.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:08.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:08.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:08.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:08.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:08.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:08.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:08.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:08.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:08.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:08.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:08.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:09.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:09.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:09.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:09.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:09.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:09.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:09.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:09.104+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:09.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:09.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:09.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:09.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:09.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:09.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:09.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:09.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:09.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:09.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:09.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:09.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:09.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:09.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:09.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:09.085+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:09.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:09.438+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44494 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:10.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:10.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:10.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:10.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:10.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:10.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:10.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:10.081+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:10.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:10.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:10.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:10.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:11.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:11.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:11.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:11.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:11.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:11.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:11.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:11.101+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:11.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:11.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:11.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:11.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:11.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:11.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:11.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:12.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:12.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:12.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:12.089+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:12.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:12.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:12.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:12.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:12.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:12.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:12.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:12.050+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:13.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:13.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:13.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:13.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:13.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:13.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:13.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:13.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:13.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:13.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:13.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:13.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:13.074+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:14.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:14.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:14.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:14.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:14.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:14.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:14.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:14.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:14.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:14.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:14.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:14.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:14.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:14.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:14.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:14.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:14.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:14.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:14.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:14.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:14.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:14.443+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44539 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:14.997+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:15.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:15.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:15.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:15.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:15.114+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:16.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:16.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:16.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:16.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:15.975+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:16.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:16.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:16.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:16.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:16.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:16.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:16.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:16.094+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:16.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:15.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:16.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:15.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:16.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:15.979+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:16.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:15.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.000+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.001+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:17.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:17.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:17.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:17.118+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:16.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:16.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:16.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:17.000+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:18.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:18.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:18.092+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:18.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:18.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:18.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:17.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:17.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:17.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:17.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:18.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:17.953+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:19.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:19.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:19.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:19.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:19.109+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:19.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:18.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:19.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:18.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:19.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:18.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:19.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:18.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:19.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:18.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:19.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:18.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:19.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:18.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:19.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:18.987+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:19.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:19.448+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44584 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:20.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:20.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:20.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:20.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:20.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:20.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:20.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:20.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:20.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:20.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:20.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:20.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:19.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:20.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:19.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:20.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:19.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:20.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:19.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:21.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:21.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:21.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:21.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:21.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:21.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:21.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:21.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:21.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:21.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:21.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:20.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:20.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:20.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:20.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:22.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:22.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:22.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:22.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:22.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:22.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:22.008+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:22.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:22.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:22.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:22.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:22.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:22.034+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:22.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:22.035+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:22.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:21.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:22.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:21.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:22.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:21.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:22.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:21.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:23.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:23.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:23.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:23.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:23.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:23.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:23.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:23.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:23.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:23.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:23.025+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:22.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:22.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:23.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:22.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:23.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:22.888+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:23.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:23.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:23.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:23.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:24.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:24.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:24.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:24.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:24.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:24.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:24.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:24.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:24.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:24.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:24.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:24.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:24.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:24.455+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44629 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:25.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:25.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:25.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:25.078+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:25.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:25.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:24.971+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:24.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:24.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:24.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:24.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:26.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:26.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:26.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:26.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:26.045+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:25.896+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:25.896+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:25.896+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:26.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:25.897+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:26.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:25.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:26.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:25.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:26.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:25.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:26.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:25.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:27.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.012+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:26.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:26.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:27.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:26.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:27.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:26.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:27.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:26.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:27.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:26.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:27.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:26.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:27.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:26.953+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:27.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:27.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:27.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:27.964+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:28.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:28.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:28.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:28.231 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:27.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:28.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:27.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:28.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:27.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:28.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:27.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:28.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:27.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:28.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:28.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:28.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:28.951+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:28.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:29.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:28.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:29.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:28.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:29.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:28.987+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:28.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:28.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:28.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:29.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:28.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:29.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:29.461+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44675 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:29.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:29.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:29.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:29.987+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:30.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:29.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:30.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:29.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:30.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:29.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:30.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:29.913+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:30.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:29.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:30.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:29.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:30.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:29.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:30.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:29.887+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:30.947+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:30.947+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:30.947+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:30.949+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:30.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:31.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:30.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:31.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:30.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:31.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:30.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:30.895+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:30.895+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:31.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:30.895+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:31.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:30.896+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:31.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:31.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:31.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:31.950+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:31.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:31.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:31.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:31.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:31.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:31.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:31.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:31.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:31.902+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:31.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:31.902+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:31.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:31.902+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:31.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:31.903+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:32.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:32.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:32.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:32.938+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:32.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:32.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:32.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:32.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:32.931+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:32.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:32.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:32.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:32.901+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:33.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:33.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:33.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:33.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:33.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:33.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:33.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:33.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:33.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:33.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:33.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:33.979+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:33.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:33.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:34.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:33.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:34.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:33.957+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:34.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:34.467+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44720 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:35.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:34.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:35.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:34.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:35.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:34.974+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:35.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:34.975+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:35.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:34.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:35.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:34.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:35.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:34.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:35.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:34.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:35.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:34.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:35.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:34.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:35.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:34.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:35.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:34.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:35.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:35.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:35.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:35.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.005+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:35.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:35.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:35.831+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:36.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:35.833+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:36.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:36.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:36.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:36.937+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:36.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:36.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:37.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:36.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:37.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:36.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:37.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:36.859+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:37.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:37.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:37.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:37.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:38.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:38.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:38.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:38.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:38.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:38.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:37.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:38.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:37.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:38.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:37.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:38.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:37.816+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:38.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:38.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:38.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:38.883+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:39.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:39.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:39.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:39.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:39.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:39.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:39.016+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:39.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:38.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:39.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:38.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:39.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:38.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:39.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:38.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:39.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:39.472+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44765 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:39.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:39.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:39.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:39.838+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:40.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:40.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:40.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:40.046+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:40.047+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:39.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:39.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:39.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:40.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:39.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:41.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:40.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:41.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:40.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:41.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:40.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:41.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:40.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:41.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:40.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:41.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:40.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:41.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:40.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:41.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:40.905+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:41.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:41.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:41.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:41.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:42.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:41.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:42.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:41.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:42.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:41.828+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:42.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:41.829+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:42.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:41.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:42.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:41.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:42.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:41.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:42.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:41.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:42.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:42.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:42.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:42.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:42.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:42.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:42.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:42.858+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:43.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:43.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:43.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:43.081+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:42.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:42.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:42.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:42.930+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:43.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:43.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:43.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:43.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:43.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:43.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:43.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:43.899+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:43.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:43.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:43.977 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:43.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:43.977 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:43.831+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:43.977 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:43.832+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:44.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:44.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:44.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:44.098+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:44.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:44.478+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44810 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:45.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:44.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:45.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:44.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:45.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:44.860+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:45.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:44.861+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:45.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:44.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:45.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:44.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:45.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:44.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:45.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:44.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:45.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:45.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:45.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:45.113+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:45.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:45.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:45.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:45.902+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:46.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:46.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:46.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:46.111+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:45.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:45.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:45.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:45.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:47.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:46.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:47.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:46.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:47.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:46.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:47.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:46.900+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:47.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:46.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:47.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:46.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:47.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:46.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:47.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:46.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:47.161+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:47.161+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:47.161+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:47.162+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:48.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:47.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:48.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:47.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:48.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:47.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:48.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:47.915+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:48.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:47.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:48.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:47.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:48.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:47.969+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:48.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:47.970+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:48.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:48.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:48.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:48.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:48.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:48.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:48.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:48.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:49.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:49.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:49.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:49.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:49.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:49.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:48.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:49.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:48.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:49.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:48.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:49.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:48.997+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:49.960 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:49.484+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44852 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:49.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:49.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:49.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:49.958+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.042+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:50.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:50.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:50.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:50.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:50.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:50.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:50.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:50.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:50.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:50.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:50.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:50.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:51.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:51.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:51.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:51.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:51.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:51.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:51.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:51.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:50.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:52.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:51.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:52.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:51.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:52.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:51.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:52.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:51.961+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:52.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:51.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:52.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:51.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:52.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:51.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:52.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:51.992+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:52.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:52.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:52.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:52.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:52.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:52.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:52.176+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:52.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:52.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:52.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:52.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:52.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:52.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:52.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:52.967+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:53.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:53.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:53.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:53.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:53.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:54.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:53.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:53.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:53.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:53.992+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:53.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:53.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:53.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:54.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:53.991+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:54.511 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:54.490+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44897 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:45:54.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:54.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:54.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:54.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:54.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:54.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:54.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:54.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:54.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:54.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:54.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:54.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:54.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:54.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:54.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:54.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:55.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:55.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:55.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:55.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:55.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:55.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:55.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:55.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:55.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:55.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:55.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:55.952+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:56.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:56.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:56.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:56.242+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:57.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:57.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:57.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:57.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:56.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:56.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:56.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:56.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:56.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:56.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:56.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:56.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:57.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:57.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:57.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:57.982+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:58.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:58.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:58.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:58.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:58.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:58.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:58.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:58.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:59.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:59.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:59.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:45:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:45:59.177+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:59.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:58.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:59.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:58.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:59.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:58.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:45:59.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:58.963+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:59.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:59.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:59.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:59.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:59.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:59.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:45:59.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:45:59.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:45:59.942 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:45:59.495+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44942 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:00.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:59.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:00.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:59.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:00.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:59.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:00.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:45:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:45:59.940+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:00.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:00.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:00.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:00.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:00.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:00.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:00.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:00.042+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:00.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:00.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:00.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:00.209+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:01.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:01.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:01.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:01.004+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:00.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:01.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:00.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:01.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:00.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:01.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:00.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:01.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:01.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:01.207+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:01.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:01.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:02.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:02.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:02.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:02.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:02.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:02.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:02.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:02.002+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:02.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:02.039+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:02.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:02.039+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:02.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:02.039+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:02.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:02.039+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:02.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:02.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:02.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:02.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:03.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:03.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:03.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:03.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:03.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:03.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:03.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:03.182+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:03.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:03.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:03.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:03.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:03.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:04.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:04.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:04.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:04.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:04.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:04.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:04.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:04.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:04.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:04.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:04.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:04.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:04.193+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:04.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:04.501+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 44987 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:05.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:05.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:05.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:05.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:05.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:05.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:05.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:05.132+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:05.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:05.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:05.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:05.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:05.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:05.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:06.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:06.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:06.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:06.168+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:06.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:06.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:06.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:06.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:06.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:06.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:06.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:06.011+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:07.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:07.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:07.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:07.128+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:07.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:07.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:07.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:07.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:07.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:07.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:08.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:08.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:08.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:08.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:08.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:08.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:08.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:08.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:08.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:07.973+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:09.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:09.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:09.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:09.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:09.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:09.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:09.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:09.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:09.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:09.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:09.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:09.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:08.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:08.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:09.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:08.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:09.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:08.999+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:09.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:09.507+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45033 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:10.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:10.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:10.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:10.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:10.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:10.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:10.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:10.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:10.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:10.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:10.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:10.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:10.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:10.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:10.038+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:11.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:11.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:11.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:11.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:11.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:11.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:11.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:11.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:11.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:11.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:11.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:11.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:11.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:11.057+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:12.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:12.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:12.159+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:12.160+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:12.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:12.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:12.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:12.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:12.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:12.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:12.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:12.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:12.094+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:13.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:13.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:13.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:13.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:13.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:13.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:13.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:13.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:13.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:13.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:13.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:13.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:14.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:14.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:14.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:14.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:14.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:14.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:14.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:14.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:14.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:14.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:14.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:14.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:14.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:14.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:14.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:14.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:14.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:14.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:14.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:14.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:14.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:14.512+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45078 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:15.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:15.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:15.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:15.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:15.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:15.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:15.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:15.042+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:15.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:15.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:15.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:15.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:15.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:16.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:16.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:16.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:16.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:16.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:16.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:16.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:16.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:16.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:16.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:16.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:16.068+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:16.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:16.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:16.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:16.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:16.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:16.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:16.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:16.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:17.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:17.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:17.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:17.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:17.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:17.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:17.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:17.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:17.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:17.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:17.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:17.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:18.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:18.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:18.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:18.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:18.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:18.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:18.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:18.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:19.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:19.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:19.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:19.225+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:18.993+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:19.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:19.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:19.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:19.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:19.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:19.518+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45122 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:20.268 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:20.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:20.269 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:20.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:20.269 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:20.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:20.269 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:20.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:20.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:19.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:19.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:19.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:19.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:20.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:20.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:20.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:20.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:20.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:21.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:21.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:21.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:21.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:21.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:21.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:21.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:21.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:21.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:21.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:21.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:21.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:21.270+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:22.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:22.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:22.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:22.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:22.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:22.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:22.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:22.064+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:22.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:22.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:22.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:22.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:22.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:23.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:23.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:23.223+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:23.224+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:23.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:23.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:23.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:23.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:23.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:23.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:23.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:23.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:23.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:23.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:23.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:23.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:24.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:24.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:24.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:24.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:24.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:24.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:24.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:24.135+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:24.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:24.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:24.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:24.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:24.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:24.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:24.524+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45168 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:25.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:25.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:25.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:25.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:25.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:25.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:25.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:25.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:25.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:25.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:25.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:25.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:26.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:26.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:26.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:26.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:26.272+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:26.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:26.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:26.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:26.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:26.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:26.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:26.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:26.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:26.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:26.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:27.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:27.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:27.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:27.252+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:27.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:27.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:27.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:27.145+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:27.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:27.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:27.050+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:27.051+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:28.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:28.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:28.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:28.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:28.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:28.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:28.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:28.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:28.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:28.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:28.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:28.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:28.173+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:29.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:29.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:29.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:29.176+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:29.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:29.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:29.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:29.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:29.979 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:29.530+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45213 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:30.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:30.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:30.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:30.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:30.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:30.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:30.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:30.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:30.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:30.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:30.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:30.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:30.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:30.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:30.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:30.201+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:30.271 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:30.271 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:30.271 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.986+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:30.271 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:29.987+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:31.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:31.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:31.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:31.111+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:31.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:30.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:31.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:30.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:30.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:30.943+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:31.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:31.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:31.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:31.251+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:32.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:31.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:32.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:31.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:32.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:31.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:32.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:31.905+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:32.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:32.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:32.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:32.112+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:32.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:32.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:32.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:32.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:33.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:33.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:33.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:33.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:33.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:33.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:33.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:33.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:33.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:32.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:33.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:32.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:33.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:32.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:33.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:32.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:33.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:33.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:33.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:33.293+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:33.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:33.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:33.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:33.858+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:34.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:34.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:34.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:34.066+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:34.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:34.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:34.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:34.293+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:34.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:34.535+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45259 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:35.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:35.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:35.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:35.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:35.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:35.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:34.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:35.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:34.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:35.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:34.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:35.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:34.888+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:35.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:35.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:35.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:35.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:35.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:36.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:35.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:36.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:35.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:36.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:35.916+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:36.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:36.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:36.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:36.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:36.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:36.071+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:36.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:36.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:36.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:36.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:36.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:36.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:36.954+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:37.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:36.955+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:37.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:37.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:37.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:37.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:37.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:37.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:37.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:37.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:37.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:37.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:37.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:38.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:38.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:38.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:38.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:37.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:37.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:37.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:38.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:37.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:38.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:38.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:38.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:38.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:39.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:39.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:39.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:39.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:39.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:38.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:39.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:38.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:39.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:38.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:39.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:38.939+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:39.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:39.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:39.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:39.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:39.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:39.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:39.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:39.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:39.668 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:39.542+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45303 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:39.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:39.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:39.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:39.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:40.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:40.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:40.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:40.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:40.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:40.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:40.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:40.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:40.319+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:40.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:40.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:40.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:41.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:40.927+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:41.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:41.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:41.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:41.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:41.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:41.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:41.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:41.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:41.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:41.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:41.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:41.302+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:42.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:42.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:42.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:42.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:41.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:41.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:41.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:41.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:42.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:42.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:42.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:42.281+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:43.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:43.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:43.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:43.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:42.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:42.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:42.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:42.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:43.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:43.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:43.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:43.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:43.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:43.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:43.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:43.293+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:43.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:43.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:43.915+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:43.916+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:44.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:44.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:44.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:44.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:44.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:44.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:44.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:44.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:44.908 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:44.547+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45349 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:44.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:44.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:44.905+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:45.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:44.906+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:45.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:45.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:45.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:45.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:45.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:45.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:45.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:45.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:45.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:46.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:46.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:46.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:46.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:46.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:46.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:46.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:46.979+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:46.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:46.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:46.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:46.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:47.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:47.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:47.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:47.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:48.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:48.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:48.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:48.021+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:47.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:47.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:47.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:47.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:48.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:48.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:48.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:48.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:48.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:48.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:48.820+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:48.821+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:49.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:49.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:49.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:49.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:49.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:49.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:49.681 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:49.552+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45394 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:49.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:49.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:49.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:49.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:49.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:49.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:49.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:49.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:49.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:49.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:49.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:49.803+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:50.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:50.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:50.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:50.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:50.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:50.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:50.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:50.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:50.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:50.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:50.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:50.342+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:50.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:50.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:50.832+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:50.833+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:51.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:51.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:51.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:51.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:51.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:51.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:51.078+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:51.584 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:51.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:51.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:51.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:51.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:51.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:51.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:51.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:51.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:51.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:51.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:51.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:52.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:52.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:52.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:52.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:52.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:52.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:52.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:52.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:53.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:53.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:53.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:53.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:52.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:52.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:53.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:52.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:53.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:52.846+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:53.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:53.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:53.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:53.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:53.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:53.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:53.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:53.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:53.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:53.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:53.823+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:54.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:54.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:54.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:54.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:54.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:54.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:54.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:54.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:54.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:54.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:54.265+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:54.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:54.558+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45440 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:54.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:54.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:54.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:54.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:54.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:54.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:54.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:54.788+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:55.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:55.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:55.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:55.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:55.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:55.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:55.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:55.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:55.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:55.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:56.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:55.749+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:56.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:55.749+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:56.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:55.749+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:56.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:55.750+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:56.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:56.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:56.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:56.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:56.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:56.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:56.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:56.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:57.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:56.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:57.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:56.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:57.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:56.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:57.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:56.768+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:57.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:57.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:57.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:57.158+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:57.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:57.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:57.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:57.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:58.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:57.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:58.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:57.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:58.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:57.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:58.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:57.780+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:58.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:58.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:58.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:58.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:58.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:58.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:58.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:58.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:59.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:58.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:59.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:58.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:59.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:58.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:59.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:58.804+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:59.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:59.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:59.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:46:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:46:59.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:46:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:59.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:59.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:59.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:46:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:46:59.253+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:59.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:46:59.564+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45484 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:46:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:59.800+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:59.800+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:59.800+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:46:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:46:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:46:59.801+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:00.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:00.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:00.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:00.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:00.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:00.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:00.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:00.251+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:01.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:00.757+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:01.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:00.757+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:01.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:00.757+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:01.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:00.758+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:01.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:01.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:01.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:01.170+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:01.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:01.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:01.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:01.271+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:02.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:01.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:02.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:01.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:02.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:01.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:02.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:01.749+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:02.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:02.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:02.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:02.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:02.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:02.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:02.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:02.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:02.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:02.135+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:03.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:02.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:03.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:02.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:03.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:02.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:03.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:02.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:03.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:03.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:03.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:03.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:03.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:03.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:03.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:03.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:03.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:03.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:03.767+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:03.768+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:04.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:04.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:04.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:04.119+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:04.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:04.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:04.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:04.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:04.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:04.570+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45530 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:04.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:04.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:04.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:04.781+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:05.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:05.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:05.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:05.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:05.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:05.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:05.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:05.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:06.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:05.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:06.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:05.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:06.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:05.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:06.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:05.808+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:06.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:06.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:06.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:06.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:06.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:06.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:06.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:06.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:06.806+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:06.806+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:06.806+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:07.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:06.807+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:07.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:07.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:07.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:07.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:07.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:07.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:07.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:07.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:07.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:07.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:08.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:07.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:08.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:07.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:08.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:07.762+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:08.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:07.764+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:08.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:08.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:08.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:08.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:08.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:08.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:08.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:08.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:09.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:08.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:09.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:08.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:09.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:08.722+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:09.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:08.723+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:09.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:09.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:09.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:09.164+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:09.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:09.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:09.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:09.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:09.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:09.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:09.576+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45575 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:09.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:09.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:09.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:09.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:10.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:10.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:10.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:10.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:10.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:10.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:10.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:10.201+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:10.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:10.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:10.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:10.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:11.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:11.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:11.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:11.111+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:11.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:11.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:11.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:11.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:11.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:11.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:11.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:11.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:12.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:12.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:12.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:12.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:12.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:12.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:12.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:12.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:12.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:12.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:12.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:12.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:13.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:13.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:13.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:13.104+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:13.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:13.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:13.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:13.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:13.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:13.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:13.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:13.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:14.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:14.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:14.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:14.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:14.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:14.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:14.200+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:14.200+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:14.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:14.582+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45621 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:14.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:14.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:14.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:14.714+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:15.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:15.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:15.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:15.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:15.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:15.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:15.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:15.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:16.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:15.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:16.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:15.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:16.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:15.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:16.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:15.738+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:16.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:16.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:16.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:16.116+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:16.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:16.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:16.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:16.176+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:16.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:16.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:16.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:16.707+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:17.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:17.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:17.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:17.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:17.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:17.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:17.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:17.220+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:17.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:17.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:17.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:17.743+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:18.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:18.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:18.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:18.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:18.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:18.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:18.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:18.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:18.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:18.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:18.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:19.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:18.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:19.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:18.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:19.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:18.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:19.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:18.736+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:19.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:19.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:19.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:19.116+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:19.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:19.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:19.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:19.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:19.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:19.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:19.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:19.588+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45665 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:19.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:19.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:19.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:19.787+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:20.095+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:20.095+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:20.095+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:20.096+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:20.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:20.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:20.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:20.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:21.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:20.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:21.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:20.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:21.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:20.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:21.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:20.770+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:21.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:21.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:21.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:21.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:21.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:21.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:21.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:21.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:22.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:21.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:22.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:21.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:22.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:21.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:22.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:21.813+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:22.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:22.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:22.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:22.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:22.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:22.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:22.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:22.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:23.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:22.851+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:23.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:22.851+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:23.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:22.851+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:23.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:22.852+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:23.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:23.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:23.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:23.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:23.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:23.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:23.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:23.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:23.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:23.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:23.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:23.820+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:24.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:24.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:24.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:24.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:24.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:24.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:24.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:24.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:24.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:24.593+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45711 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:25.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:25.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:25.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:25.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:25.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:25.086+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:25.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:25.087+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:25.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:24.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:25.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:24.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:25.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:24.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:25.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:24.847+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:25.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:25.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:25.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:25.130+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:26.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:25.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:26.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:25.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:26.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:25.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:26.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:25.800+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:26.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:26.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:26.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:26.158+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:26.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:26.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:26.117+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:26.118+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:27.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:26.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:27.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:26.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:27.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:26.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:27.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:26.830+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:27.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:27.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:27.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:27.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:27.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:27.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:27.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:27.090+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:27.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:27.091+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:28.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:28.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:28.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:28.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:28.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:28.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:28.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:28.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:28.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:28.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:28.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:28.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:28.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:28.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:28.079+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:28.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:27.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:28.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:27.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:28.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:27.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:28.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:27.868+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:29.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:29.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:29.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:29.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:29.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:29.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:29.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:29.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:29.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:28.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:29.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:28.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:29.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:28.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:29.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:28.852+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:29.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:29.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:29.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:29.123+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:29.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:29.599+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45756 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:30.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:30.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:30.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:30.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:30.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:30.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:30.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:30.099+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:30.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:29.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:30.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:29.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:30.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:29.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:30.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:29.853+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:31.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:30.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:31.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:30.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:31.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:30.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:31.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:30.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:31.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:31.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:31.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:31.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:31.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:31.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:31.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:31.122+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:32.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:31.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:32.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:31.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:32.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:31.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:32.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:31.784+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:32.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:32.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:32.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:32.167+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:32.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:32.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:32.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:32.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:33.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:32.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:33.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:32.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:33.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:32.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:33.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:32.785+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:33.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:33.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:33.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:33.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:33.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:33.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:33.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:33.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:33.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:33.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:33.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:33.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:33.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:33.772+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:33.773+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:34.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:34.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:34.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:34.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:34.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:34.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:34.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:34.188+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:34.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:34.604+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45802 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:34.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:34.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:34.779+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:34.780+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:35.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:35.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:35.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:35.102+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:35.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:35.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:35.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:35.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:35.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:35.187+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:36.071 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:35.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:36.071 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:35.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:36.071 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:35.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:36.072 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:35.795+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:36.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:36.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:36.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:36.068+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:36.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:36.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:36.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:36.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:37.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:37.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:37.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:37.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:37.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:37.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:37.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:37.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:37.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:36.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:37.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:36.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:37.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:36.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:37.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:36.842+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:37.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:37.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:37.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:37.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:38.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:38.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:38.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:38.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:38.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:38.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:38.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:38.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:38.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:37.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:38.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:37.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:38.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:37.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:38.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:37.846+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:38.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:38.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:38.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:38.109+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:39.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:39.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:39.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:39.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:39.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:39.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:39.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:39.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:39.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:38.851+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:39.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:38.851+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:39.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:38.851+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:39.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:38.852+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:39.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:39.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:39.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:39.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:39.902 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:39.610+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45846 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:40.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:40.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:40.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:40.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:40.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:40.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:40.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:40.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:40.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:39.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:40.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:39.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:40.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:39.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:40.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:39.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:40.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:40.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:40.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:40.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:41.015 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:40.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:41.015 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:40.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:41.016 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:40.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:41.016 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:40.857+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:41.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:41.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:41.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:41.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:41.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:41.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:41.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:41.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:41.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:41.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:41.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:41.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:42.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:42.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:42.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:42.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:42.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:42.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:42.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:42.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:42.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:42.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:42.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:42.890+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:42.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:43.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:43.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:43.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:43.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:43.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:43.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:43.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:43.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:43.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:43.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:43.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:43.925+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:43.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:44.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:44.882 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:44.617+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45892 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:44.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:44.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:44.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:44.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:44.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:44.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:44.880+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:44.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:44.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:44.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:44.978+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:45.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:45.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:45.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:45.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:45.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:45.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:45.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:45.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:45.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:45.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:45.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:45.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:46.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:46.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:46.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:46.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:47.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:47.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:47.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:47.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:46.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:46.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:46.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:46.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:47.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:47.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:47.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:47.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:47.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:47.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:48.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:48.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:48.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:48.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:47.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:47.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:47.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:47.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:48.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:48.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:48.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:48.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:49.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:49.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:49.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:49.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:49.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:49.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:49.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:49.070+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:49.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:48.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:49.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:48.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:49.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:48.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:49.266 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:49.000+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:49.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:49.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:49.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:49.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:49.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:49.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:49.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:49.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:49.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:49.623+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45937 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:50.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:50.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:50.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:50.219+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:50.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:50.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:50.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:50.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:50.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:50.112+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:50.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:50.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:50.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:50.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:51.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:51.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:51.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:51.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:51.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:51.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:51.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:51.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:51.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:51.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:51.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:51.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:52.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:52.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:52.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:52.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:52.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:52.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:52.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:52.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:52.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:52.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:52.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:52.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:52.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:53.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:53.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:53.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:53.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:53.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:53.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:53.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:53.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:53.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:53.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:53.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:53.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:53.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:54.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:54.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:54.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:54.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:54.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:54.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:54.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:54.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:54.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:54.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:54.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:54.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:54.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:54.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:54.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:54.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:54.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:54.629+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 45984 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:47:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:55.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:55.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:55.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:55.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:55.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:55.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:55.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:55.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:55.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:55.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:55.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:55.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:55.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:55.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:55.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:55.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:55.212+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:56.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:56.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:56.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:56.018+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:56.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:56.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:56.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:56.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:56.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:56.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:56.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:56.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:56.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:56.146+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:57.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:57.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:57.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:57.191+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:57.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:57.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:57.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:57.151+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:57.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:57.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:57.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:57.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:57.042+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:58.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:58.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:58.226+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:58.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:58.135+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:58.135+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:58.135+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:58.136+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:58.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:58.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:58.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:58.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:58.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:59.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:59.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:59.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:47:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:47:59.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:59.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:59.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:59.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:47:59.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:47:59.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:59.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:59.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:59.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:59.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:59.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:59.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:47:59.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:47:59.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:47:59.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:47:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:47:59.634+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46028 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:00.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:00.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:00.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:00.277+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:00.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:00.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:00.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:00.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:00.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:00.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:00.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:00.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:00.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:00.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:01.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:01.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:01.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:01.007+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:01.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:01.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:01.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:01.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:01.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:01.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:01.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:01.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:02.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:02.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:02.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:02.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:02.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:02.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:02.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:02.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:02.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:02.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:02.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:02.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:02.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:02.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:02.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:02.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:03.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:03.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:03.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:03.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:03.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:03.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:03.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:03.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:03.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:03.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:03.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:03.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:03.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:03.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:03.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:03.125+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:03.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:03.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:03.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:03.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:03.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:03.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:03.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:03.345+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:04.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:04.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:04.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:04.004+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:04.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:04.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:04.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:04.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:04.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:04.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:04.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:04.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:04.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:04.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:04.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:04.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:04.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:04.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:04.641+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46074 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:05.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:05.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:05.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:05.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:05.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:05.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:05.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:05.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:05.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:05.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:05.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:05.073+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:05.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:05.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:05.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:05.362+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:06.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:06.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:06.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:06.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:05.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:06.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:06.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:06.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:06.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:06.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:06.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:06.335 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:06.092+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:06.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:06.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:06.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:06.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:07.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:07.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:07.307 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:07.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:07.307 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:07.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:07.307 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:07.026+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:07.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:07.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:07.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:07.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:07.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:07.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:07.308 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:07.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:07.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:07.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:07.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:07.304+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:08.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:08.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:08.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:08.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:08.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:08.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:08.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:08.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:08.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:08.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:08.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.027+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:08.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.028+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:08.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:08.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:08.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:08.323+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:09.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:09.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:09.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:09.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:08.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:09.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:09.135+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:09.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:09.135+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:09.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:09.135+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:09.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:09.136+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:09.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:09.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:09.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:09.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:09.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:09.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:09.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:09.297+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:09.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:09.646+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46119 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:10.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:09.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:10.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:09.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:10.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:09.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:10.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:09.986+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:10.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:10.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:10.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:10.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:10.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:10.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:10.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:10.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:10.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:10.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:10.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:10.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:11.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:11.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:11.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:11.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:11.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:11.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:11.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:11.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:11.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:11.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:11.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:11.092+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:11.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:11.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:11.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:11.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:11.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:12.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:12.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:12.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:12.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:12.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:12.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:12.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:12.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:12.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:12.348+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:12.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:12.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:12.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:12.960+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:13.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:13.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:13.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:13.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:13.090+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:13.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:13.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:13.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:13.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:13.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:13.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:13.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:13.362+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:13.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:13.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:13.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:13.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:13.928+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:14.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:14.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:14.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:14.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:14.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:14.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:14.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:14.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:14.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:14.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:14.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:14.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:14.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:14.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:14.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:14.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:14.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:14.652+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46165 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:14.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:14.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:14.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:14.972+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:15.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:15.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:15.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:15.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:15.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:15.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:15.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:15.090+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:15.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:15.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:15.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:15.288+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:16.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:16.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:16.020+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:16.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:16.021+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:16.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:16.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:16.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:16.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:16.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:16.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:16.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:16.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:16.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:16.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:16.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:16.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:16.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:17.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:17.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:17.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:17.062+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:17.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:17.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:17.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:17.274+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:17.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:17.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:17.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:17.092+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:18.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:18.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:18.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:18.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:18.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:18.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:18.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:18.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:18.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:18.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:18.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:18.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:19.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:19.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:19.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:19.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:19.013+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:19.013+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:19.013+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:19.014+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:19.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:19.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:19.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:19.064+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:19.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:19.658+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46209 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:20.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:20.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:20.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:20.233+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:20.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:20.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:20.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:20.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:20.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:20.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:20.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:20.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:20.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:20.089+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:21.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:21.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:21.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:21.130+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.025+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.026+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:21.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:21.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:21.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:21.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:21.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:22.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:22.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:22.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:22.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:21.992+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:22.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:22.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:22.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:22.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:22.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:23.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:23.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:23.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:23.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:22.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:22.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:22.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:22.945+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:23.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:23.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:23.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:23.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:23.077+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:23.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:23.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:23.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:23.984+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:24.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:24.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:24.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:24.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:24.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:24.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:24.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:24.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:24.948 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:24.664+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46255 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:24.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:24.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:24.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:24.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:25.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:25.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:25.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:25.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:25.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:25.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:25.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:25.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:25.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:25.103+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:25.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:25.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:26.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:25.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:26.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:25.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:26.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:25.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:26.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:25.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:26.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:26.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:26.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:26.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:26.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:26.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:26.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:26.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:26.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:26.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:26.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:26.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:26.215+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:26.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:26.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:26.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:26.962+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:27.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:27.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:27.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:27.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:27.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:27.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:27.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:27.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:27.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:27.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:27.233+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:27.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:27.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:27.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:27.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:27.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:27.990+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:28.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:28.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:28.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:28.237+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:28.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:28.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:28.076+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:28.077+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:28.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:28.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:28.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:28.944+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:29.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:29.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:29.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:29.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:29.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:29.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:29.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:29.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:29.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:29.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:29.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:29.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:29.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:29.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:29.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:29.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:29.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:29.670+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46300 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:30.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:29.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:30.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:29.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:30.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:29.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:30.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:29.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:30.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:30.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:30.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:30.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:30.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:30.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:30.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:30.038+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:30.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:30.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:30.321+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:30.322+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:30.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:30.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:30.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:30.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:31.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:31.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:31.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:31.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:31.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:31.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:31.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:31.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:31.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:31.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:31.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:31.290+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:32.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:32.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:32.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:32.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:32.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:32.006+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:32.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:32.007+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:32.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:32.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:32.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:32.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:32.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:32.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:32.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:32.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:33.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:33.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:33.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:33.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:33.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:33.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:33.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:33.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:33.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:33.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:33.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:33.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:33.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:33.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:33.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:33.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:33.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:33.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:33.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:33.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:33.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:34.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:34.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:34.336 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:34.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:34.336 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:34.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:34.336 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:34.048+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:34.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:34.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:34.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:34.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:34.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:34.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:34.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:34.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:34.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:34.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:34.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:34.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:34.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:34.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:34.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:34.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:34.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:34.676+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46346 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:35.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:35.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:35.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:35.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:35.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:35.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:35.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:35.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:35.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:35.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:35.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:35.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:35.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:35.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:35.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:35.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:35.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:35.158+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:36.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:36.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:36.126+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:36.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:36.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:36.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:36.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:36.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:36.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:36.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:36.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:36.062+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:36.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:36.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:36.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:36.287+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:37.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:37.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:37.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:37.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:37.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:37.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:37.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:37.320+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:37.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:37.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:37.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:37.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:37.175+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:38.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:38.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:38.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:38.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:38.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:38.106+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:38.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:38.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:38.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:38.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:38.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:38.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:38.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:38.315+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:38.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:38.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:38.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:38.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:38.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:38.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:38.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:38.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:39.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:39.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:39.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:39.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:39.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:39.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:39.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:39.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:39.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:39.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:39.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:39.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:39.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:39.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:39.234+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:39.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:39.682+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46390 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:40.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:40.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:40.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:40.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:40.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:40.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:40.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:40.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:40.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:40.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:40.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:40.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:40.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:40.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:40.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:40.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:40.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:40.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:40.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:41.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:41.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:41.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:41.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:41.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:41.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:41.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:41.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:41.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:41.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:41.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:41.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:41.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:42.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:42.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:42.149+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:42.150+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:42.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:42.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:42.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:42.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:42.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:42.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:42.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:42.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:42.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:42.265+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:43.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:43.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:43.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:43.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:43.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:43.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:43.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:43.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:43.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:43.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:43.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:43.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:43.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:43.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:43.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:43.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:43.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:44.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:44.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:44.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:44.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:44.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:44.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:44.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:44.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:44.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:44.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:44.178+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:44.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:44.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:44.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:44.688+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46436 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:45.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:45.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:45.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:45.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:45.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:45.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:45.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:45.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:45.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:45.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:45.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:45.162+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:45.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:45.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:45.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:45.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:45.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:45.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:45.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:46.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:46.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:46.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:46.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:46.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:46.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:46.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:46.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:46.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:46.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:46.160+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:47.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:47.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:47.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:47.306+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:47.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:47.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:47.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:47.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:47.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:47.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:47.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:47.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:47.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:47.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:47.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:47.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:47.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:47.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:47.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:48.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:48.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:48.158+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:48.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:48.159+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:48.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:48.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:48.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:48.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:48.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:48.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:48.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:48.095+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:48.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:48.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:48.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:48.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:48.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:48.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:48.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:48.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:49.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:49.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:49.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:49.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:49.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:49.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:49.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:49.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:49.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:49.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:49.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:49.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:49.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:49.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:49.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:49.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:49.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:49.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:49.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:49.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:49.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:49.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:49.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:49.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:49.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:49.694+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46481 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:50.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:50.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:50.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:50.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:50.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:50.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:50.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:50.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:50.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:50.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:50.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:50.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:50.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:50.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:50.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:50.127+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:50.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:50.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:50.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:50.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:51.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:51.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:51.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:51.014+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:51.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:51.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:51.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:51.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:51.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:51.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:51.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:51.384+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:52.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:52.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:52.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:52.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:52.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:52.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:52.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:52.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:52.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:52.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:52.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:52.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:52.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:52.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:52.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:52.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:52.397+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:52.397+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:52.397+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:52.398+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:53.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:53.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:53.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:53.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:53.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:53.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:53.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:53.066+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:53.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:53.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:53.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:53.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:53.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:53.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:53.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:53.125+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:53.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:53.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:53.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:53.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:53.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:53.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:53.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:53.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:54.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:54.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:54.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:54.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:54.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:54.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:54.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:54.153+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:54.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:54.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:54.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:54.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:54.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:54.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:54.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:54.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:54.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:54.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:54.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:54.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:54.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:54.699+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46527 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:48:55.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:55.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:55.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:55.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:55.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:55.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:55.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:55.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:55.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:55.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:55.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:55.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:55.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:55.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:55.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:55.153+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:55.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:55.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:55.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:55.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:56.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:56.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:56.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:56.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:56.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:56.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:56.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:56.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:56.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:56.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:56.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:56.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:56.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:56.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:56.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:56.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:56.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:56.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:56.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:56.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:57.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:57.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:57.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:57.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:57.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:57.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:57.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:57.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:57.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:57.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:57.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:57.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:57.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:57.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:57.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:57.095+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:57.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:57.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:57.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:57.387+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:58.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:58.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:58.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:58.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:58.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:58.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:58.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:58.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:58.361 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:58.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:58.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:58.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:58.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:58.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:58.362 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:58.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:58.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:58.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:58.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:58.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:59.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:59.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:59.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:59.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:59.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:59.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:48:59.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:48:59.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:59.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:59.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:59.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:59.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:59.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:59.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:48:59.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:48:59.039+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:59.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:59.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:59.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:59.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:59.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:59.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:48:59.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:48:59.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:48:59.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:48:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:48:59.706+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46571 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:00.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:00.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:00.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:00.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:00.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:00.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:00.083+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:00.084+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:00.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:00.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:00.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:00.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:00.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:01.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:01.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:01.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:01.116+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:01.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:01.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:01.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:01.043+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:01.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:01.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:01.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:01.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:02.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:02.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:02.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:02.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:02.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:02.029+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:02.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:02.029+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:02.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:02.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:02.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:02.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:02.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:02.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:02.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:02.298+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:03.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:03.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:03.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:03.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:03.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:03.017+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:03.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:03.017+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:03.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:03.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:03.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:03.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:03.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:03.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:03.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:03.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:03.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:03.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:03.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:03.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:03.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:03.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:03.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:03.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:04.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:04.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:04.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:04.052+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:04.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:04.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:04.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:04.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:04.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:04.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:04.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:04.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:04.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:04.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:04.712+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46617 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:05.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:05.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:05.084+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:05.085+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:05.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:05.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:05.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:05.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:05.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:05.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:05.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:05.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:05.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:05.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:05.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:05.199+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:06.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:06.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:06.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:06.114+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:06.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:06.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:06.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:06.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:06.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:06.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:06.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:06.199+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:07.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:07.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:07.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:07.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:07.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:07.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:07.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:07.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:07.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:07.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:07.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:07.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:07.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:08.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:08.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:08.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:08.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:08.182+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:08.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:08.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:08.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:08.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:08.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:08.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:08.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:08.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:08.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:09.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:09.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:09.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:09.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:09.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:09.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:09.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:09.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:09.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:09.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:09.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:09.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:09.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:09.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:09.287+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:09.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:09.718+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46662 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:10.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:10.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:10.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:10.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:10.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:10.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:10.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:10.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:10.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:10.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:10.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:10.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:11.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:11.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:11.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:11.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:11.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:11.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:11.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:11.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:11.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:11.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:11.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:11.311+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:12.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:12.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:12.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:12.200+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:12.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:12.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:12.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:12.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:12.243+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:12.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:12.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:12.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:12.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:12.350+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:13.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:13.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:13.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:13.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:13.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:13.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:13.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:13.199+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:13.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:13.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:13.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:13.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:13.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:13.314+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:14.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:14.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:14.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:14.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:14.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:14.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:14.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:14.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:14.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:14.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:14.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:14.273+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:15.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:14.723+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46708 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:15.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:15.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:15.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:15.318+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:15.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:15.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:15.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:15.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:15.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:15.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:15.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:15.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:15.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:16.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:16.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:16.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:16.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:16.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:16.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:16.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:16.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:16.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:16.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:16.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:16.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:16.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:16.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:16.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:17.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:17.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:17.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:17.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:17.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:17.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:17.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:17.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:17.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:17.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:17.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:17.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:18.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:18.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:18.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:18.174+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:18.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:18.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:18.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:18.302+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:18.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:18.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:18.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:18.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:18.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:19.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:19.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:19.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:19.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:19.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:19.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:19.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:19.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:19.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:19.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:19.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:19.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:19.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:19.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:20.159 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:19.729+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46752 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:20.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:20.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:20.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:20.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:20.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:20.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:20.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:20.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:20.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:20.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:20.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:20.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:20.382+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:21.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:21.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:21.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:21.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:21.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:21.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:21.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:21.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:21.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:21.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:21.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:21.176+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:22.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:22.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:22.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:22.327+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:22.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:22.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:22.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:22.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:22.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:22.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:22.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:22.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:23.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:23.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:23.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:23.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:23.177+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:23.177+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:23.177+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:23.178+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:23.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:23.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:23.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:23.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:23.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:23.302+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:24.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:24.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:24.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:24.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:24.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:24.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:24.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:24.182+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:24.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:24.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:24.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:24.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:24.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:25.136 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:24.735+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46798 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:25.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:25.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:25.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:25.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:25.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:25.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:25.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:25.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:25.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:25.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:25.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:25.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:25.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:25.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:25.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:25.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:25.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:25.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:26.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:26.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:26.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:26.338+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:26.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:26.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:26.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:26.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:26.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:26.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:26.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:26.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:26.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:26.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:27.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:27.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:27.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:27.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:27.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:27.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:27.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:27.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:27.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:27.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:27.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:27.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:27.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:27.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:27.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:27.306+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:27.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:27.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:27.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:27.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:28.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:28.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:28.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:28.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:28.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:28.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:28.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:28.058+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:28.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:28.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:28.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:28.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:28.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:28.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:28.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:28.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:29.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:29.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:29.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:29.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:29.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:29.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:29.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:29.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:29.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:29.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:29.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:29.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:29.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:29.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:30.138 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:29.740+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46843 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:30.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:30.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:30.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:30.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:30.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:30.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:30.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:30.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:30.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:30.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:30.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:30.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:30.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:30.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:30.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:30.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:30.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:30.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:30.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:30.419+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:31.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:31.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:31.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:31.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:31.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:31.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:31.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:31.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:31.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:31.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:31.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:31.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:31.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:31.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:31.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:31.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:31.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:31.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:31.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:31.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:32.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:32.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:32.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:32.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:32.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:32.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:32.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:32.336+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:32.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:32.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:32.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:32.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:32.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:32.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:32.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:32.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:32.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:32.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:32.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:32.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:33.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:33.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:33.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:33.312+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:33.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:33.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:33.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:33.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:33.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:33.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:33.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:33.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:34.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:34.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:34.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:34.317+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:34.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:34.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:34.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:34.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:34.768 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:34.746+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46889 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:34.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:34.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:34.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:34.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:34.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:34.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:34.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:34.507+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:35.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:35.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:35.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:35.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:35.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:35.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:35.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:35.124+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:35.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:35.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:35.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:35.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:35.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:35.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:35.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:35.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:35.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:35.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:35.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:35.466+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:36.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:36.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:36.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:36.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:36.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:36.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:36.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:36.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:36.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:36.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:36.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:36.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:36.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:37.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:37.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:37.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:37.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:37.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:37.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:37.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:37.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:37.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:37.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:37.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:37.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:38.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:38.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:38.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:38.361+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:38.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:38.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:38.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:38.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:38.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:38.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:38.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:38.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:39.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:39.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:39.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:39.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:39.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:39.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:39.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:39.203+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:39.493 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:39.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:39.493 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:39.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:39.493 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:39.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:39.493 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:39.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:39.493 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:39.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:39.493 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:39.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:39.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:39.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:39.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:39.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:40.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:39.752+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46933 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:40.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:40.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:40.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:40.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:40.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:40.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:40.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:40.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:40.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:40.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:40.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:40.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:40.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:40.250+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:40.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:40.250+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:40.546 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:40.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:40.546 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:40.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:40.546 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:40.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:40.546 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:40.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:41.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:41.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:41.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:41.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:41.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:41.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:41.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:41.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:41.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:41.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:41.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:41.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:41.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:41.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:41.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:41.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:41.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:41.460+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:42.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:42.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:42.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:42.204+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:42.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:42.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:42.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:42.488+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:42.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:42.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:42.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:42.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:42.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:42.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:42.502+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:43.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:43.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:43.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:43.165+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:43.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:43.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:43.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:43.500+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:43.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:43.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:43.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:43.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:43.534+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:44.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:44.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:44.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:44.214+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:44.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:44.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:44.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:44.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:44.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:44.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:44.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:44.486+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:44.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:44.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:44.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:44.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:44.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:44.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:44.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:44.525+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:45.200 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:44.758+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 46979 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:45.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:45.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:45.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:45.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:45.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:45.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:45.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:45.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:45.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:45.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:45.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:45.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:45.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:45.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:45.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:45.500+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:46.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:46.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:46.163+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:46.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:46.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:46.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:46.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:46.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:46.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:46.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:46.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:46.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:46.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:46.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:46.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:46.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:46.538+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:47.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:47.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:47.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:47.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:47.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:47.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:47.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:47.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:47.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:47.484+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:47.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:47.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:47.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:47.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:47.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:48.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:48.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:48.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:48.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:48.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:48.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:48.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:48.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:48.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:48.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:48.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:48.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:48.559+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:49.489 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:49.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:49.489 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:49.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:49.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:49.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:49.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:49.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:49.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:49.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:49.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:49.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:49.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:49.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:49.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:49.486+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:49.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:49.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:49.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:49.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:49.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:49.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:49.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:49.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:50.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:49.763+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47024 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:50.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:50.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:50.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:50.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:50.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:50.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:50.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:50.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:50.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:50.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:50.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:50.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:50.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:50.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:50.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:50.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:50.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:50.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:50.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:50.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:51.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:51.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:51.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:51.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:51.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:51.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:51.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:51.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:51.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:51.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:51.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:51.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:51.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:51.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:51.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:51.627+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:52.583 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:52.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:52.583 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:52.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:52.583 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:52.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:52.583 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:52.277+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:52.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:52.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:52.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:52.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:52.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:52.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:52.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:52.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:52.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:52.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:52.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:52.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:53.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:53.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:53.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:53.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:53.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:53.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:53.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:53.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:53.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:53.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:53.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:53.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:53.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:53.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:53.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:53.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:53.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:53.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:53.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:53.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:54.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:54.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:54.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:54.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:54.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:54.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:54.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:54.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:54.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:54.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:54.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:54.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:54.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:54.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:54.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:54.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:54.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:54.770+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47069 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:54.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:54.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:54.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:54.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:54.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:54.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:54.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:54.611+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:55.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:55.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:55.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:55.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:55.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:55.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:55.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:55.437+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:55.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:55.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:55.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:55.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:55.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:55.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:55.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:55.266+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:55.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:55.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:55.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:55.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:55.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:55.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:55.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:55.627+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:56.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:56.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:56.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:56.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:56.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:56.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:56.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:56.270+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:56.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:56.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:56.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:56.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:56.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:56.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:56.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:56.451+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:56.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:56.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:56.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:56.600+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:57.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:57.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:57.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:57.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:57.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:57.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:57.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:57.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:57.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:57.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:57.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:57.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:57.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:57.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:57.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:57.319+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:57.632+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:57.632+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:57.632+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:57.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:58.618 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:58.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:58.618 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:58.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:58.618 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:58.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:58.618 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:58.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:58.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:58.329+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:58.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:58.329+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:58.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:58.329+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:58.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:58.330+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:58.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:58.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:58.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:58.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:59.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:59.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:59.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:59.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:59.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:59.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:49:59.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:49:59.454+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:49:59.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:59.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:59.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:59.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:59.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:59.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:49:59.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:49:59.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:59.900 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:49:59.776+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47114 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:49:59.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:59.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:59.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:59.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:59.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:59.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:49:59.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:49:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:49:59.600+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:00.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:00.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:00.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:00.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:00.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:00.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:00.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:00.582 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:00.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:00.582 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:00.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:00.582 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:00.324+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:00.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:00.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:00.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:00.578+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:01.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:01.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:01.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:01.319+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:01.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:01.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:01.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:01.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:01.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:01.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:01.725 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:01.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:01.725 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:01.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:01.725 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:01.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:02.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:02.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:02.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:02.568+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:02.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:02.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:02.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:02.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:02.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:02.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:02.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:02.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:02.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:02.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:02.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:02.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:02.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:02.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:02.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:03.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:03.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:03.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:03.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:03.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:03.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:03.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:03.315+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:03.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:03.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:03.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:03.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:03.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:03.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:03.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:03.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:03.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:03.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:03.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:03.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:03.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:03.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:03.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:03.533+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:04.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:04.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:04.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:04.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:04.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:04.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:04.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:04.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:04.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:04.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:04.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:04.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:04.726 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:04.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:04.726 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:04.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:04.726 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:04.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:04.726 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:04.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:04.726 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:04.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:04.726 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:04.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:05.120 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:04.782+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47159 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:05.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:05.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:05.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:05.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:05.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:05.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:05.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:05.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:05.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:05.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:05.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:05.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:05.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:05.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:05.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:05.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:05.495+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:06.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:06.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:06.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:06.320+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:06.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:06.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:06.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:06.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:06.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:06.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:06.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:06.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:06.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:06.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:06.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:06.517+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:07.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:07.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:07.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:07.293+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:07.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:07.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:07.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:07.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:07.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:07.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:07.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:07.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:07.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:07.506+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:08.304+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:08.304+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:08.304+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:08.305+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:08.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:08.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:08.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:08.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:08.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:08.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:08.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:08.545+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:09.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:09.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:09.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:09.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:09.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:09.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:09.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:09.515+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:09.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:09.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:09.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:09.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:10.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:09.789+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47205 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:10.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:10.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:10.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:10.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:10.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:10.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:10.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:10.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:10.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:10.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:10.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:10.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:10.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:10.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:10.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:10.351+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:10.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:10.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:10.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:10.560+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:11.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:11.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:11.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:11.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:11.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:11.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:11.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:11.322+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:11.590 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:11.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:11.590 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:11.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:11.590 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:11.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:11.590 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:11.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:11.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:11.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:11.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:11.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:12.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:12.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:12.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:12.560+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:12.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:12.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:12.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:12.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:12.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:12.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:12.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:12.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:12.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:12.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:12.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:12.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:12.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:13.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:13.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:13.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:13.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:13.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:13.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:13.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:13.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:13.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:13.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:13.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:13.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:13.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:13.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:13.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:13.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:13.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:13.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:13.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:13.589+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:14.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:14.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:14.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:14.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:14.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:14.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:14.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:14.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:14.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:14.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:14.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:14.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:14.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:14.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:14.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:14.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:14.918 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:14.795+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47250 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:14.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:14.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:14.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:14.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:14.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:14.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:14.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:14.609+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:15.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:15.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:15.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:15.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:15.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:15.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:15.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:15.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:15.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:15.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:15.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:15.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:15.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:15.398+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:15.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:15.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:15.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:15.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:16.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:16.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:16.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:16.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:16.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:16.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:16.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:16.235+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:16.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:16.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:16.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:16.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:16.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:16.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:16.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:16.350+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:16.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:16.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:16.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:16.651+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:17.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:17.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:17.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:17.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:17.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:17.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:17.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:17.211+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:17.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:17.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:17.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:17.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:18.600 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:18.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:18.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:18.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:18.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:18.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:18.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:18.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:18.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:18.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:18.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:18.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:18.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:18.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:18.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:18.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:18.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:18.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:18.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:18.598+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:19.329+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:19.329+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:19.329+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:19.330+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:19.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:19.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:19.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:19.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:19.924 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:19.801+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47295 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:19.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:19.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:19.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:19.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:19.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:19.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:19.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:19.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:20.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:20.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:20.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:20.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:20.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:20.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:20.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:20.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:20.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:20.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:20.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:20.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:20.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:20.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:21.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:21.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:21.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:21.212+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:21.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:21.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:21.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:21.293+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:21.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:21.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:21.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:21.653+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:22.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:22.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:22.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:22.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:22.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:22.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:22.610 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:22.235+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:22.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:22.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:22.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:22.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:22.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:22.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:22.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:22.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:22.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:22.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:22.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:22.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:23.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:23.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:23.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:23.200+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:23.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:23.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:23.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:23.324+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:23.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:23.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:23.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:23.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:24.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:24.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:24.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:24.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:24.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:24.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:24.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:24.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:24.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:24.932 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:24.808+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47340 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:24.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:24.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:24.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:24.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:24.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:24.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:24.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:24.623+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:25.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:25.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:25.158+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:25.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:25.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:25.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:25.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:25.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:25.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:25.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:25.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:25.393+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:25.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:25.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:25.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:25.645+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:26.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:26.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:26.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:26.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:26.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:26.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:26.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:26.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:26.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:26.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:26.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:26.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:26.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:26.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:26.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:26.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:26.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:27.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:27.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:27.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:27.168+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:27.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:27.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:27.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:27.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:27.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:27.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:27.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:27.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:27.621+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:27.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:28.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:28.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:28.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:28.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:28.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:28.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:28.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:28.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:28.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:28.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:28.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:28.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:28.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:28.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:28.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:28.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:29.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:29.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:29.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:29.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:29.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:29.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:29.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:29.224+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:29.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:29.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:29.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:29.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:29.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:29.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:29.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:29.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:29.938 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:29.814+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47387 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:29.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:29.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:29.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:29.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:29.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:29.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:29.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:29.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:30.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:30.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:30.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:30.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:30.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:30.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:30.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:30.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:30.495+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:30.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:30.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:30.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:30.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:30.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:30.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:31.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:31.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:31.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:31.197+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:31.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:31.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:31.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:31.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:31.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:31.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:31.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:31.602+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:31.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:31.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:31.826 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:31.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:31.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:31.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:31.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:31.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:32.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:32.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:32.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:32.178+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:32.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:32.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:32.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:32.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:32.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:32.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:32.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:32.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:32.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:33.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:33.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:33.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:33.190+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:33.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:33.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:33.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:33.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:33.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:33.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:33.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:33.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:33.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:33.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:33.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:33.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:34.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:34.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:34.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:34.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:34.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:34.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:34.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:34.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:34.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:34.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:34.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:34.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:34.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:34.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:34.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:34.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:34.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:34.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:35.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:34.821+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47432 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:35.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:35.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:35.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:35.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:35.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:35.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:35.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:35.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:35.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:35.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:35.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:35.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:35.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:35.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:35.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:35.545+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:35.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:35.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:35.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:35.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:35.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:35.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:35.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:35.614+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:36.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:36.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:36.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:36.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:36.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:36.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:36.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:36.253+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:36.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:36.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:36.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:36.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:36.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:36.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:36.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:36.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:36.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:36.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:37.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:37.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:37.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:37.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:37.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:37.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:37.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:37.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:37.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:37.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:37.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:37.591+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:37.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:37.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:37.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:37.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:37.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:38.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:38.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:38.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:38.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:38.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:38.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:38.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:38.242+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:38.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:38.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:38.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:38.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:38.611+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:38.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:38.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:38.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:38.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:38.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:38.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:38.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:38.578+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:39.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:39.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:39.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:39.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:39.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:39.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:39.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:39.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:39.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:39.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:39.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:39.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:39.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:39.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:39.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:39.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:39.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:39.565+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:39.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:39.565+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:39.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:39.565+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:39.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:39.566+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:40.209 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:39.827+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47477 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:40.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:40.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:40.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:40.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:40.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:40.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:40.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:40.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:40.557+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:40.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:40.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:40.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:40.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:40.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:40.536+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:41.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:41.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:41.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:41.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:41.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:41.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:41.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:41.242+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:41.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:41.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:41.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:41.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:41.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:41.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:41.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:41.552+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:41.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:41.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:41.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:41.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:41.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:41.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:41.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:41.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:42.533 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:42.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:42.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:42.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:42.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:42.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:42.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:42.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:42.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:42.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:42.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:42.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:42.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:42.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:43.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:43.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:43.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:43.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:43.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:43.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:43.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:43.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:43.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:43.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:43.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:43.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:43.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:43.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:43.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:43.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:43.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:43.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:43.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:43.576+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:44.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:44.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:44.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:44.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:44.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:44.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:44.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:44.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:44.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:44.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:44.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:44.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:44.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:44.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:44.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:44.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:44.836 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:44.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:44.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:44.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:44.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:44.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:44.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:44.576+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:45.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:44.832+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47522 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:45.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:45.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:45.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:45.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:45.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:45.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:45.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:45.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:45.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:45.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:45.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:45.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:45.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:45.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:45.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:45.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:45.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:45.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:45.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:45.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:45.620+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:46.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:46.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:46.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:46.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:46.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:46.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:46.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:46.277+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:46.577 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:46.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:46.577 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:46.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:46.578 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:46.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:46.578 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:46.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:46.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:46.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:46.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:46.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:46.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:46.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:46.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:46.573+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:47.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:47.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:47.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:47.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:47.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:47.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:47.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:47.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:47.914 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:47.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:47.914 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:47.492+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:47.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:47.560+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:47.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:47.560+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:47.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:47.560+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:47.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:47.561+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:48.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:48.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:48.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:48.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:48.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:48.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:48.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:48.306+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:48.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:48.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:48.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:48.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:48.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:48.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:48.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:48.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:48.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:48.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:48.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:48.611+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:49.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:49.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:49.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:49.569+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:49.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:49.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:49.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:49.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:49.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:49.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:49.357+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:49.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:49.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:49.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:49.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:49.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:50.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:49.839+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47564 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:50.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:50.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:50.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:50.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:50.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:50.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:50.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:50.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:50.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:50.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:50.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:50.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:50.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:50.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:50.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:50.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:50.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:50.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:50.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:50.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:51.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:51.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:51.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:51.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:51.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:51.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:51.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:51.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:51.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:51.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:51.419+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:51.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:51.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:51.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:51.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:51.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:51.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:51.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:51.506+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:52.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:52.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:52.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:52.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:52.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:52.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:52.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:52.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:52.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:52.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:52.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:52.510+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:53.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:53.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:53.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:53.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:53.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:53.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:53.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:53.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:53.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:53.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:53.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:53.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:53.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:53.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:53.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:53.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:53.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:53.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:53.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:53.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:53.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:53.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:53.717 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:53.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:54.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:54.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:54.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:54.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:54.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:54.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:54.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:54.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:54.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:54.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:54.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:54.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:54.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:54.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:54.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:54.317+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:54.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:54.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:54.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:54.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:54.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:54.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:54.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:54.572+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:55.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:54.845+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47609 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:55.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:55.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:55.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:55.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:55.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:55.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:55.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:55.309+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:55.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:55.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:55.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:55.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:55.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:55.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:55.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:55.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:55.566+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:55.566+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:55.566+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:55.567+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:56.559 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:56.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:56.559 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:56.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:56.559 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:56.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:56.559 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:56.487+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:56.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:56.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:56.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:56.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:56.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:56.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:56.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:56.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:56.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:56.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:56.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:56.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:57.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:57.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:57.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:57.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:57.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:57.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:57.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:57.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:57.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:57.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:57.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:57.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:57.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:57.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:57.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:57.268+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:57.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:57.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:57.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:57.573+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:58.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:58.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:58.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:58.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:58.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:58.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:58.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:58.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:58.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:58.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:58.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:58.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:58.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:58.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:58.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:58.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:58.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:58.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:58.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:59.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:59.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:59.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:59.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:59.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:59.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:50:59.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:50:59.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:50:59.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:59.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:59.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:59.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:59.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:59.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:50:59.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:50:59.427+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:59.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:50:59.851+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47655 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:50:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:59.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:59.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:59.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:50:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:50:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:50:59.630+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:00.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:00.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:00.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:00.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:00.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:00.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:00.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:00.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:00.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:00.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:00.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:00.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:00.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:00.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:00.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:00.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:01.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:01.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:01.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:01.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:01.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:01.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:01.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:01.368+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:01.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:01.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:01.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:01.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:01.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:01.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:01.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:01.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:01.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:01.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:01.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:01.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:01.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:02.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:02.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:02.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:02.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:02.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:02.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:02.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:02.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:02.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:02.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:02.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:02.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:02.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:02.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:02.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:02.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:02.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:02.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:02.620+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:02.621+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:03.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:03.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:03.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:03.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:03.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:03.297+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:03.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:03.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:03.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:03.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:03.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:03.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:03.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:03.354+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:03.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:03.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:03.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:03.654+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:04.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:04.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:04.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:04.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:04.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:04.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:04.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:04.352+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:04.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:04.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:04.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:04.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:04.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:04.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:04.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:04.280+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:04.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:04.858+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47699 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:04.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:04.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:04.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:04.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:05.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:05.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:05.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:05.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:05.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:05.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:05.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:05.277+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:05.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:05.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:05.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:05.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:05.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:05.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:05.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:05.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:05.686+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:05.686+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:05.686+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:05.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:06.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:06.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:06.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:06.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:06.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:06.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:06.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:06.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:06.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:06.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:06.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:06.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:06.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:06.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:06.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:06.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:06.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:06.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:06.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:06.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:06.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:06.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:06.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:06.726+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:07.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:07.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:07.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:07.270+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:07.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:07.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:07.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:07.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:07.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:07.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:07.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:07.312+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:07.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:07.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:07.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:07.772+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:08.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:08.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:08.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:08.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:08.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:08.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:08.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:08.265+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:08.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:08.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:08.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:08.753+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:09.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:09.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:09.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:09.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:09.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:09.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:09.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:09.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:09.243+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:10.213 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:09.863+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47746 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:10.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:09.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:10.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:09.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:10.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:09.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:10.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:09.802+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:10.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:10.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:10.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:10.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:10.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:10.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:10.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:10.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:10.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:11.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:10.782+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:11.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:10.782+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:11.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:10.782+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:11.202 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:10.783+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:11.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:11.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:11.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:11.348+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:11.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:11.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:11.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:11.199+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:11.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:11.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:11.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:11.830+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:12.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:12.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:12.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:12.243+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:12.244+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:12.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:12.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:12.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:12.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:12.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:12.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:12.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:12.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:12.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:13.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:13.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:13.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:13.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:13.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:13.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:13.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:13.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:13.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:13.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:13.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:13.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:13.860+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:14.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:14.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:14.362+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:14.363+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:14.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:14.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:14.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:14.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:14.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:15.186 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:14.869+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47790 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:15.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:14.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:15.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:14.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:15.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:14.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:15.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:14.887+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:15.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:15.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:15.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:15.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:15.379+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:15.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:15.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:15.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:15.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:15.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:16.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:15.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:16.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:15.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:16.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:15.843+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:16.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:16.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:16.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:16.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:16.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:16.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:16.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:16.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:16.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:17.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:16.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:17.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:16.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:17.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:16.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:17.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:16.804+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:17.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:17.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:17.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:17.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:17.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:17.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:17.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:17.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:17.188+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:18.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:17.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:18.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:17.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:18.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:17.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:18.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:17.845+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:18.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:18.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:18.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:18.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:18.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:18.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:18.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:18.172+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:18.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:18.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:19.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:18.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:19.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:18.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:19.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:18.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:19.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:18.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:19.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:19.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:19.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:19.366+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:19.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:19.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:19.164+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:19.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:19.164+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:20.122 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:19.876+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47836 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:20.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:19.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:20.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:19.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:20.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:19.796+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:20.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:19.797+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:20.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:20.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:20.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:20.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:20.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:20.320+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:21.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:20.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:21.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:20.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:21.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:20.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:21.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:20.799+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:21.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:21.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:21.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:21.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:21.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:21.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:21.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:21.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:21.121+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:22.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:21.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:22.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:21.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:22.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:21.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:22.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:21.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:22.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:22.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:22.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:22.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:22.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:22.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:22.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:22.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:22.090+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:23.058 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:22.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:23.058 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:22.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:23.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:22.763+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:23.059 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:22.763+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:23.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:23.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:23.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:23.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:23.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:23.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:23.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:23.056+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:23.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:23.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:23.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:23.751+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:24.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:24.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:24.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:24.241+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:24.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:24.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:24.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:24.019+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:25.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:24.882+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47880 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:24.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:24.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:24.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:24.772+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:25.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:25.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:25.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:25.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:25.039+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:25.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:25.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:25.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:25.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:26.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:25.807+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:26.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:25.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:26.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:25.808+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:26.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:25.808+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:26.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:26.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:26.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:26.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:26.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:26.032+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:26.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:26.033+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:26.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:26.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:26.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:26.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:27.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:26.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:27.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:26.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:27.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:26.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:27.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:26.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:27.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:27.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:27.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:27.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:27.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:27.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:27.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:27.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:27.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:27.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:27.755+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:27.756+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:28.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:28.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:28.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:28.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:28.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:28.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:28.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:28.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:28.031+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:29.028 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:28.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:29.029 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:28.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:29.029 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:28.771+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:29.029 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:28.772+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:29.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:29.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:29.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:29.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:29.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:29.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:29.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:29.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:29.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:30.143 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:29.889+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47927 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:30.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:30.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:30.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:30.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:30.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:30.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:30.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:30.001+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:30.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:29.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:30.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:29.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:30.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:29.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:30.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:29.764+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:30.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:30.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:30.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:30.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:30.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:31.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:31.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:31.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:31.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:31.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:31.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:31.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:31.049+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:31.053 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:30.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:31.053 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:30.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:31.053 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:30.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:31.053 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:30.799+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:31.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:31.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:31.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:31.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:32.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:31.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:32.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:31.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:32.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:31.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:32.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:31.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:32.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:32.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:32.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:32.127+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:32.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:32.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:32.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:32.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:32.061+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:33.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:32.783+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:33.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:32.783+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:33.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:32.783+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:33.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:32.784+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:33.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:33.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:33.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:33.110+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:33.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:33.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:33.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:33.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:33.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:33.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:33.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:33.815+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:33.816+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:34.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:34.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:34.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:34.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:34.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:34.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:34.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:34.101+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:35.122 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:34.895+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 47971 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:35.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:35.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:35.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:35.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:35.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:35.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:35.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:35.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:35.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:34.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:35.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:34.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:35.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:34.865+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:35.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:34.866+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:35.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:35.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:35.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:35.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:36.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:35.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:36.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:35.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:36.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:35.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:36.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:35.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:36.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:36.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:36.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:36.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:36.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:36.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:36.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:36.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:36.095+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:37.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:36.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:37.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:36.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:37.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:36.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:37.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:36.802+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:37.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:37.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:37.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:37.176+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:37.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:37.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:37.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:37.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:37.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:37.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:37.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:37.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:38.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:38.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:38.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:38.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:38.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:38.073+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:38.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:38.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:38.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:37.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:38.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:37.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:38.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:37.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:38.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:37.850+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:38.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:38.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:38.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:38.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:39.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:39.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:39.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:39.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:39.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:39.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:39.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:39.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:39.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:38.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:39.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:38.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:39.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:38.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:39.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:38.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:39.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:39.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:39.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:39.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:40.205 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:39.901+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48017 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:40.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:39.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:40.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:39.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:40.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:39.870+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:40.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:39.871+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:40.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:40.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:40.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:40.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:40.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:40.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:40.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:40.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:40.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:40.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:40.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:40.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:41.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:41.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:41.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:41.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:41.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:41.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:41.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:41.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:41.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:40.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:41.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:40.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:41.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:40.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:41.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:40.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:41.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:41.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:41.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:41.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:42.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:42.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:42.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:42.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:41.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:41.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:41.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:42.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:41.939+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:42.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:42.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:42.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:42.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:43.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:43.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:43.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:43.092+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:42.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:42.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:42.914+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:42.915+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:43.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:43.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:43.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:43.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:43.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:43.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:43.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:43.242+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:43.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:43.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:43.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:43.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:44.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:44.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:44.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:44.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:44.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:44.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:44.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:44.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:44.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:44.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:45.209 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:44.908+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48061 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:45.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:44.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:45.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:44.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:45.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:44.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:45.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:44.857+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:45.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:45.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:45.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:45.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:45.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:45.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:45.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:45.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:45.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:45.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:45.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:45.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:46.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:46.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:46.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:46.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:45.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:45.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:45.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:45.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:46.251+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:46.251+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:46.251+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:46.252+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:47.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:47.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:47.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:47.057+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:46.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:46.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:46.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:46.851+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:47.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:47.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:47.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:47.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:47.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:47.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:47.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:47.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:47.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:47.895+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:48.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:48.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:48.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:48.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:48.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:48.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:48.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:48.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:48.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:48.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:48.225+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:48.226+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:49.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:49.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:49.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:49.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:49.064+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:48.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:48.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:48.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:49.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:48.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:49.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:49.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:49.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:49.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:49.186+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:50.223 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:49.914+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48108 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:50.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:50.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:50.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:50.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:50.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:49.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:49.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:49.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:50.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:49.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:50.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:50.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:50.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:50.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.001+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:50.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:50.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:51.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:50.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:51.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:50.954+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:51.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:51.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:51.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:51.234+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:52.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:51.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:52.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:51.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:52.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:51.937+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:52.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:51.938+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:52.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:52.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:52.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:52.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:51.955+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:52.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:52.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:52.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:52.203+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:52.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:52.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:52.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:52.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:52.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:52.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:52.985+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:52.986+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:53.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:53.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:53.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:53.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:53.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:53.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:53.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:53.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:53.954+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:53.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:53.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:53.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:53.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:53.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:53.961+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:54.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:54.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:54.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:54.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:54.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:54.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:54.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:54.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:54.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:54.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:54.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:54.952+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:55.224 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:54.920+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48152 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:51:55.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:55.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:55.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:55.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:55.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:55.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:55.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:55.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:55.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:55.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:55.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:55.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:55.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:55.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:55.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:55.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:55.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:55.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:55.954+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:56.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:56.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:56.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:56.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:56.902+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:56.902+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:56.902+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:56.902+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.002+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:57.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:57.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:57.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:57.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:57.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:57.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:57.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:57.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:57.958+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:58.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:58.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:58.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:58.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:58.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:58.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:58.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:51:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:58.954+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:58.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:58.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:58.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:51:59.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:58.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:51:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:59.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:59.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:59.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:51:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:51:59.385+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:00.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:51:59.926+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48198 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:59.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:59.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:59.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:51:59.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:59.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:59.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:59.927+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:51:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:51:59.929+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:00.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:00.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:00.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:00.425+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:00.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:00.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:00.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:00.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:00.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:01.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:00.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:01.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:00.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:01.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:00.962+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:01.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:01.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:01.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:01.453+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:02.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:02.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:02.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:02.002+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:01.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:01.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:02.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:01.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:02.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:01.945+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:02.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:02.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:02.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:02.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:02.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:02.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:02.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:02.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:02.916+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:03.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:03.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:03.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:03.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:03.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:03.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:03.050+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:03.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:03.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:03.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:03.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:03.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:03.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:03.909+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:03.909+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:03.909+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:03.909+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:04.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:04.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:04.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:04.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:04.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:04.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:04.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:04.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:04.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:05.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:04.932+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48242 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:04.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:04.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:04.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:04.908+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:05.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:05.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:05.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:05.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:05.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:05.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:05.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:05.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:05.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:05.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:05.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:05.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:05.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:05.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:05.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:05.871+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:06.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:06.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:06.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:06.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:06.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:06.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:06.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:06.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:06.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:06.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:06.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:06.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:06.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:06.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:06.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:06.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:06.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:06.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:06.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:06.877+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:07.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:07.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:07.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:07.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:07.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:07.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:07.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:07.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:07.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:07.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:07.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:07.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:07.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:07.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:08.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:07.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:08.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:07.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:08.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:07.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:08.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:07.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:08.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:08.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:08.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:08.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:08.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:08.111+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:08.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:08.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:08.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:08.455+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:09.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:08.840+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:09.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:08.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:09.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:08.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:09.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:08.841+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:09.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:09.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:09.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:09.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:09.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:09.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:09.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:09.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:09.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:09.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:09.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:41.569250+0000 (oldest deadline 2024-02-20T08:42:01.569250+0000) 2024-02-20T08:52:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:09.461+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:10.063 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:09.938+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48289 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:10.063 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:09.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:10.063 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:09.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:10.063 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:09.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:37.519166+0000 (oldest deadline 2024-02-20T08:41:57.519166+0000) 2024-02-20T08:52:10.064 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:09.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:10.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:10.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:10.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:10.165+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:10.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:10.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:11.072 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:10.773+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:11.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:11.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:11.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:11.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:11.779 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:11.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:12.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:11.776+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:12.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:12.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:12.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:12.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:12.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:12.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:13.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:12.779+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:13.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:13.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:13.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:13.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:13.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:13.732+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:13.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:14.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:14.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:14.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:14.157+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:14.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:14.718+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:14.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:14.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:15.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:14.944+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48333 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:15.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:15.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:15.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:41:39.918674+0000 (oldest deadline 2024-02-20T08:41:59.918674+0000) 2024-02-20T08:52:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:15.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:15.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:15.577+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:16.180+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:16.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:16.603+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:17.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:17.665+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:17.620+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:18.164+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:18.676+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:18.572+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:19.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:19.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:19.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:19.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:19.561+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:20.207 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:19.950+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48379 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:20.204+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:20.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:20.633+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:20.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:20.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:21.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:21.241+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:21.661+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:21.644+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:22.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:22.249+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:22.677+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:22.603+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:23.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:23.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:23.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:23.612+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:24.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:24.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:24.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:24.708+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:24.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:24.626+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:25.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:24.957+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48423 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:25.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:25.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:25.653+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:25.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:25.712+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:26.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:26.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:26.726+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:26.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:27.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:27.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:27.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:27.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:28.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:28.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:28.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:28.689+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:28.674+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:29.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:29.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:29.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:29.659+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:29.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:29.630+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:30.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:29.962+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48470 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:30.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:30.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:30.674+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:30.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:30.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:30.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:30.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:30.671+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:31.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:31.221+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:31.721+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:31.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:31.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:31.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:31.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:31.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:32.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:32.231+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:32.706+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:32.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:32.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:32.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:32.621+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:33.207+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:33.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:33.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:33.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:33.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:33.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:33.582+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:34.168+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:34.573+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:34.573+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:34.573+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:34.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:34.574+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:34.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:34.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:34.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:34.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:35.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:34.969+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48514 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:35.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:35.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:35.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:35.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:35.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:35.590+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:35.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:35.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:35.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:35.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:35.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:35.740+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:36.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:36.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:36.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:36.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:36.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:36.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:36.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:36.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:36.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:36.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:36.758+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:37.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:37.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:37.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:37.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:37.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:37.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:37.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:37.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:37.528+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:38.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:37.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:37.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:37.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:37.806+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:38.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:38.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:38.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:38.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:38.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:38.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:38.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:38.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:38.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:38.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:38.792+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:39.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:39.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:39.041+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:39.042+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:39.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:39.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:39.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:39.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:39.596+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:39.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:39.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:39.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:39.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:39.796+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:40.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:39.975+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48560 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:40.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:40.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:40.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:40.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:40.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:40.060+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:40.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:40.061+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:40.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:40.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:40.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:40.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:40.644+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:40.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:40.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:40.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:40.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:40.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:40.821+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:40.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:40.822+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:41.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:41.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:41.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:41.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:41.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:41.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:41.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:41.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:41.678+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:41.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:41.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:41.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:41.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:41.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:41.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:41.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:41.807+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:42.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:42.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:42.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:42.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:42.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:42.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:42.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:42.647+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:42.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:42.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:42.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:42.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:42.843+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:43.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:43.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:43.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:43.055+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:43.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:43.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:43.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:43.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:43.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:43.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:43.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:43.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:43.795+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:43.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:43.796+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.012+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:44.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:44.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:44.647+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:44.648+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:44.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:44.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:44.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:44.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:44.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:44.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:44.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:44.776+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:45.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:44.982+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48604 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:44.978+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:45.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:45.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:45.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:45.647+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:45.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:45.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:45.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:45.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:45.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:45.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:45.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:45.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:45.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:45.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:45.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:45.973+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:46.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:46.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:46.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:46.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:46.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:46.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:46.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:46.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:46.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:46.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:46.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:46.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:46.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:46.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:46.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:46.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:46.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:46.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:46.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:46.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:47.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:47.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:47.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:47.693+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:47.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:47.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:47.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:47.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:47.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:47.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:47.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:47.643+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:47.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:47.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:47.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:47.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:48.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:48.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:48.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:48.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:48.596+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:48.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:48.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:48.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:48.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:48.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:48.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:48.728+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:49.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:49.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:49.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:49.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:49.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:49.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:49.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:49.730+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:49.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:49.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:49.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:49.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:49.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:49.623+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:49.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:49.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:50.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:49.988+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48651 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:50.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:50.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:50.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:50.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:50.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:50.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:50.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:50.041+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:50.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:50.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:50.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:50.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:50.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:50.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:50.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:50.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:50.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:50.686+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:51.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:51.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:51.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:51.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:51.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:51.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:51.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:51.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:51.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:51.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:51.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:51.685+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:51.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:51.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:51.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:51.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:51.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:51.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:51.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:51.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:52.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:52.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:52.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:52.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:52.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:52.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:52.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:52.687+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:52.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:52.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:52.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:52.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:52.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:52.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:52.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:52.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:53.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:53.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:53.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:53.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:53.694+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:54.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:53.694+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:54.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:53.694+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:54.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:53.694+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:54.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:53.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:54.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:53.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:54.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:53.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:54.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:53.742+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:54.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:54.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:54.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:54.126+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:54.998 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:54.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:54.998 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:54.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:54.998 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:54.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:54.998 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:54.722+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:54.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:54.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:54.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:54.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:54.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:54.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:54.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:54.782+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:55.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:52:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:52:54.994+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48695 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:52:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:55.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:55.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:55.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:55.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:55.165+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:56.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:55.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:56.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:55.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:56.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:55.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:56.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:55.753+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:56.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:55.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:56.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:55.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:56.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:55.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:56.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:55.754+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:56.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:56.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:56.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:56.199+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:56.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:56.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:57.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:56.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:57.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:56.741+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:57.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:56.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:57.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:56.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:57.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:56.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:57.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:56.745+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:57.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:57.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:57.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:57.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:57.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:57.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:57.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:57.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:57.709+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:58.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:57.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:58.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:57.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:58.125 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:57.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:58.125 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:57.784+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:58.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:58.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:58.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:58.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:59.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:58.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:59.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:58.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:59.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:58.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:52:59.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:58.817+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:52:59.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:58.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:59.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:58.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:59.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:58.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:52:59.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:58.729+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:59.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:59.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:59.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:52:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:52:59.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:00.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:00.001+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48742 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:00.013 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:59.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:00.013 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:59.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:00.013 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:59.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:00.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:52:59.794+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:00.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:59.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:00.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:59.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:00.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:59.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:00.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:52:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:52:59.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:00.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:00.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:00.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:00.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:00.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:00.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:00.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:00.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:01.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:00.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:01.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:00.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:01.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:00.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:01.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:00.817+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:01.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:00.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:01.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:00.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:01.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:00.722+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:01.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:00.723+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:01.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:01.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:01.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:01.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:02.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:01.813+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:02.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:01.813+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:02.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:01.813+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:02.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:01.814+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:02.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:01.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:02.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:01.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:02.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:01.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:02.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:01.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:02.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:02.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:02.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:02.176+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:02.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:02.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:02.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:02.799+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:02.739+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:02.739+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:03.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:02.739+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:03.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:02.740+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:03.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:03.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:03.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:03.221+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:04.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:03.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:04.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:03.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:04.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:03.774+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:04.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:03.775+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:04.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:03.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:04.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:03.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:04.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:03.807+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:04.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:03.807+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:04.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:04.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:04.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:04.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:04.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:04.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:04.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:05.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:05.007+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48786 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:05.029 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:04.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:05.029 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:04.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:05.029 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:04.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:05.029 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:04.769+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:05.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:04.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:05.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:04.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:05.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:04.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:05.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:04.772+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:05.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:05.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:05.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:05.181+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:05.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:05.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:05.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:05.778+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:06.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:05.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:06.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:05.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:06.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:05.763+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:06.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:05.763+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:06.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:06.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:06.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:06.196+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:06.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:06.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:06.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:06.715+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:06.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:06.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:06.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:06.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:06.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:06.748+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:07.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:07.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:07.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:07.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:07.221+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:07.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:07.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:07.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:07.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:07.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:07.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:07.672+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:07.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:07.672+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:07.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:07.672+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:07.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:07.673+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:08.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:08.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:08.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:08.235+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:08.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:08.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:08.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:08.666+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:08.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:08.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:08.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:08.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:08.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:08.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:09.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:09.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:09.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:09.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:09.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:09.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:09.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:09.283+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:09.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:09.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:09.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:09.658+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:09.617+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:09.617+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:09.617+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:09.618+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:10.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:10.013+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48833 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:10.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:10.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:10.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:10.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:10.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:10.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:10.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:10.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:10.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:10.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:10.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:10.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:10.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:10.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:10.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:10.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:10.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:10.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:10.655+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:11.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:11.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:11.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:11.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:11.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:11.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:11.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:11.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:11.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:11.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:11.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:11.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:11.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:11.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:11.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:11.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:11.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:11.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:11.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:11.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:11.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:11.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:11.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:11.577+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:12.600 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:12.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:12.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:12.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:12.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:12.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:12.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:12.273+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:12.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:12.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:12.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:12.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:12.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:12.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:12.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:12.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:12.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:12.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:13.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:13.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:13.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:13.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:13.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:13.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:13.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:13.237+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:13.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:13.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:13.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:13.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:13.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:13.720+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:13.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:13.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:13.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:13.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:13.551+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:14.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:14.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:14.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:14.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:14.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:14.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:14.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:14.247+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:14.975 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:14.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:14.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:14.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:14.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:14.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:14.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:14.726+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:14.976 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:14.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:14.976 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:14.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:14.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:14.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:14.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:14.576+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:15.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:15.019+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48877 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:15.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:15.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:15.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:15.246+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:15.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:15.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:15.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:15.753+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:15.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:15.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:15.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:15.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:15.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:15.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:16.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:16.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:16.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:16.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:16.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:16.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:16.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:16.281+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:16.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:16.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:16.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:16.775+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:16.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:16.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:16.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:16.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:16.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:16.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:17.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:17.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:17.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:17.588 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:17.283+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:17.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:17.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:17.803+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:17.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:17.803+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:17.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:17.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:17.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:17.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:17.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:17.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:18.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:18.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:18.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:18.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:18.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:18.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:18.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:18.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:18.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:18.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:18.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:18.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:18.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:18.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:18.578+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:19.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:19.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:19.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:19.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:19.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:19.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:19.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:19.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:19.814 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:19.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:19.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:19.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:19.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:19.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:19.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:19.551+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:19.815 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:19.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:19.816 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:19.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:19.816 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:19.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:19.816 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:19.812+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:20.148 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:20.025+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48923 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:20.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:20.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:20.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:20.233+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:20.857 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:20.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:20.857 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:20.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:20.857 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:20.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:20.857 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:20.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:20.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:20.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:20.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:20.854+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:21.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:21.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:21.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:21.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:21.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:21.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:21.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:21.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:21.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:21.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:21.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:21.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:21.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:21.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:21.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:21.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:21.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:21.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:21.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:21.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:22.493 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:22.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:22.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:22.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:22.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:22.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:22.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:22.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:22.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:22.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:22.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:22.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:22.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:22.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:22.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:22.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:22.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:22.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:22.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:22.783+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:23.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:23.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:23.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:23.453+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:23.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:23.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:23.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:23.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:23.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:23.231+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:24.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:23.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:24.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:23.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:24.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:23.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:24.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:23.744+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:24.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:24.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:24.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:24.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:24.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:24.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:24.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:24.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:24.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:24.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:24.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:24.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:24.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:24.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:24.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:24.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:24.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:24.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:24.700+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:24.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:24.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:25.449 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:25.030+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 48967 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:25.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:25.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:25.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:25.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:25.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:25.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:25.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:25.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:25.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:25.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:25.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:25.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:25.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:25.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:25.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:25.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:25.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:26.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:26.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:26.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:26.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:26.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:26.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:26.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:26.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:26.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:26.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:26.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:26.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:26.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:26.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:26.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:26.439+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:26.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:26.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:26.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:26.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:26.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:27.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:27.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:27.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:27.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:27.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:27.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:27.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:27.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:27.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:27.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:27.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:27.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:27.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:27.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:27.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:27.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:27.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:27.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:27.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:27.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:27.664+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:28.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:28.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:28.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:28.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:28.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:28.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:28.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:28.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:28.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:28.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:28.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:28.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:28.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:28.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:28.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:28.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:28.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:28.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:28.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:28.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:28.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:28.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:28.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:28.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:29.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:29.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:29.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:29.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:29.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:29.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:29.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:29.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:29.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:29.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:29.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:29.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:29.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:29.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:29.680+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:29.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:29.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:29.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:29.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:29.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:29.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:30.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:30.037+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49014 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:30.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:30.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:30.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:30.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:30.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:30.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:30.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:30.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:30.944 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:30.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:30.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:30.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:30.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:30.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:30.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:30.687+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:30.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:30.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:30.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:30.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:30.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:30.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:30.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:30.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:31.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:31.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:31.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:31.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:31.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:31.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:31.546 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:31.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:31.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:31.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:31.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:31.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:31.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:31.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:31.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:31.652+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:31.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:31.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:31.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:31.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:31.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:31.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:31.838 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:31.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:32.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:32.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:32.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:32.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:32.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:32.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:32.577 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:32.314+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:32.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:32.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:32.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:32.574+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:32.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:32.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:32.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:32.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:32.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:32.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:33.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:33.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:33.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:33.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:33.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:33.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:33.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:33.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:33.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:33.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:33.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:33.711+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:33.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:33.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:33.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:33.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:33.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:33.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:33.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:33.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:34.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:34.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:34.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:34.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:34.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:34.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:34.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:34.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:35.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:34.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:34.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:34.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:34.726+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:35.325 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:35.043+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49059 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:35.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:35.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:35.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:35.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:35.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:35.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:35.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:35.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:36.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:35.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:35.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:35.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:35.751+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:36.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:36.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:36.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:36.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:36.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:36.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:36.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:36.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:36.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:36.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:36.289+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:36.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:36.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:36.729+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:36.730+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:37.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:37.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:37.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:37.523+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:37.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:37.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:37.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:37.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:37.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:37.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:37.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:37.329+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:37.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:37.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:37.767+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:37.768+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:38.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:38.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:38.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:38.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:38.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:38.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:38.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:38.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:38.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:38.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:38.775+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:38.775+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:39.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:39.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:39.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:39.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:39.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:39.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:39.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:39.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:40.073 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:40.049+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49104 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:40.073 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:39.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:40.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:39.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:40.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:39.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:40.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:39.813+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:40.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:40.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:40.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:40.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:40.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:40.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:40.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:40.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:40.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:40.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:40.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:40.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:40.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:40.799+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:40.799+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:40.799+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:41.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:40.799+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:41.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:41.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:41.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:41.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:41.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:41.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:41.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:41.252+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:41.787+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:41.787+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:41.787+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:41.788+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:42.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:42.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:42.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:42.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:42.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:42.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:42.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:42.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:42.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:42.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:42.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:42.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:42.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:42.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:42.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:42.828+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:43.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:43.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:43.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:43.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:43.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:43.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:43.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:43.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:43.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:43.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:43.832+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:44.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:43.832+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:44.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:43.832+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:44.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:43.834+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:44.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:44.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:44.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:44.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:44.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:44.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:44.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:44.329+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:45.179 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:45.056+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49149 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:45.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:44.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:45.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:44.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:45.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:44.849+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:45.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:44.850+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:45.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:45.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:45.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:45.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:45.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:45.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:45.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:45.313+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:45.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:45.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:45.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:45.848+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:46.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:46.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:46.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:46.396+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:46.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:46.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:46.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:46.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:46.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:46.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:46.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:46.817+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:47.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:47.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:47.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:47.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:47.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:47.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:47.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:47.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:47.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:47.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:47.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:47.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:47.820+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:48.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:47.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:48.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:48.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:48.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:48.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:48.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:48.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:48.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:48.341+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:48.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:48.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:48.798+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:48.799+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:49.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:49.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:49.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:49.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:49.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:49.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:49.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:49.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:49.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:49.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:50.186 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:50.062+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49195 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:50.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:49.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:50.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:49.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:50.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:49.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:50.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:49.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:50.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:50.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:50.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:50.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:50.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:50.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:50.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:50.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:50.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:50.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:50.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:50.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:51.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:51.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:51.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:51.326+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:51.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:51.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:51.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:51.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:51.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:51.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:51.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:51.842+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:52.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:52.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:52.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:52.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:52.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:52.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:52.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:52.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:52.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:52.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:52.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:52.835+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:53.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:53.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:53.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:53.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:53.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:53.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:53.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:53.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:53.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:53.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:53.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:53.828+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:54.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:54.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:54.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:54.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:54.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:54.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:54.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:54.363+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:55.093 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:53:55.068+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49240 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:53:55.093 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:54.836+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:55.093 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:54.836+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:55.094 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:54.836+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:55.094 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:54.837+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:55.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:55.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:55.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:55.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:55.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:55.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:55.304+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:55.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:55.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:55.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:55.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:55.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:55.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:55.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:55.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:55.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:55.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:55.876+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:55.876+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:56.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:56.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:56.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:56.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:56.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:56.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:56.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:56.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:56.850+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:56.850+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:56.850+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:56.850+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:57.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:57.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:57.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:57.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:57.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:57.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:57.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:57.357+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:57.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:57.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:57.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:57.807+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:58.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:58.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:58.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:58.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:58.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:58.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:58.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:58.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:58.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:58.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:58.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:58.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:58.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:53:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:58.836+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:53:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:59.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:59.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:59.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:53:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:53:59.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:59.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:59.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:59.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:59.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:59.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:59.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:53:59.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:53:59.373+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:00.077 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:59.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:00.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:59.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:00.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:59.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:00.078 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:53:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:53:59.793+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:00.393 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:00.074+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49285 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:00.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:00.266+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:00.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:00.266+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:00.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:00.266+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:00.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:00.266+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:00.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:00.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:00.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:00.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:00.749+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:00.749+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:00.749+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:00.749+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:01.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:01.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:01.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:01.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:01.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:01.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:01.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:01.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:01.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:01.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:01.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:01.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:01.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:01.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:01.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:01.351+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:01.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:01.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:01.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:01.772+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:02.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:02.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:02.193+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:02.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:02.194+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:02.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:02.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:02.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:02.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:02.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:03.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:02.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:03.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:02.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:03.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:02.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:03.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:02.777+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:03.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:03.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:03.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:03.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:03.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:03.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:03.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:03.416+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:04.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:03.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:04.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:03.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:04.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:03.816+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:04.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:03.817+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:04.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:04.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:04.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:04.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:04.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:04.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:04.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:04.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:04.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:05.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:04.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:05.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:04.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:05.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:04.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:05.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:04.857+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:05.344 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:05.080+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49330 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:05.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:05.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:05.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:05.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:05.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:05.160+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:05.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:05.161+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:05.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:05.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:05.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:05.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:05.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:05.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:05.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:05.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:05.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:05.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:05.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:06.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:05.848+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:06.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:06.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:06.195+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:06.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:06.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:06.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:06.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:06.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:06.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:06.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:06.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:06.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:06.894+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:07.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:07.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:07.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:07.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:07.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:07.235+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:07.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:07.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:07.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:07.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:07.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:08.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:07.851+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:08.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:07.851+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:08.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:07.851+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:08.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:07.852+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:08.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:08.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:08.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:08.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:08.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:08.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:08.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:08.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:08.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:08.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:08.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:08.193+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:08.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:08.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:08.824+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:08.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:09.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:09.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:09.222+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:09.223+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:09.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:09.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:09.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:09.266+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:10.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:09.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:10.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:09.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:10.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:09.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:10.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:09.822+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:10.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:10.087+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49376 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:10.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:10.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:10.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:10.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:10.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:10.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:10.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:10.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:10.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:10.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:10.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:10.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:10.801+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:11.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:11.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:11.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:11.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:11.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:11.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:11.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:11.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:11.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:11.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:11.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:11.773+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:12.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:12.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:12.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:12.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:12.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:12.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:12.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:12.287+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:12.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:12.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:12.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:12.793+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:13.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:13.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:13.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:13.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:13.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:13.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:13.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:13.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:13.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:13.252+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:14.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:13.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:14.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:13.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:14.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:13.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:14.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:13.816+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:14.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:14.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:14.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:14.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:14.214+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:14.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:14.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:14.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:14.248+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:15.218 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:15.093+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49421 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:15.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:14.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:15.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:14.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:15.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:14.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:15.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:14.867+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:15.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:15.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:15.248+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:15.249+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:15.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:15.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:15.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:15.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:15.292+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:15.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:15.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:15.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:16.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:15.910+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:16.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:16.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:16.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:16.242+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:16.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:16.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:16.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:16.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:16.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:16.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:17.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:16.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:17.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:16.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:17.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:16.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:17.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:16.862+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:17.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:17.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:17.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:17.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:17.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:17.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:17.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:17.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:17.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:17.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:17.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:17.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:17.868+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:18.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:18.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:18.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:18.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:18.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:18.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:18.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:18.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:19.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:18.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:19.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:18.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:19.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:18.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:19.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:18.831+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:19.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:19.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:19.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:19.201+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:19.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:19.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:19.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:19.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:20.113 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:20.099+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49466 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:20.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:19.854+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:20.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:19.854+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:20.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:19.854+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:20.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:19.855+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:20.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:20.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:20.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:20.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:20.212+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:20.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:20.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:20.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:20.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:20.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:21.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:20.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:21.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:20.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:21.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:20.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:21.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:20.898+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:21.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:21.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:21.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:21.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:21.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:21.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:21.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:21.197+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:22.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:21.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:22.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:21.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:22.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:21.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:22.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:21.907+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:22.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:22.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:22.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:22.137+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:22.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:22.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:22.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:22.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:22.173+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:23.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:22.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:23.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:22.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:23.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:22.889+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:23.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:22.890+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:23.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:23.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:23.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:23.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:23.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:23.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:23.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:23.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:24.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:23.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:24.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:23.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:24.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:23.890+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:24.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:23.891+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:24.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:24.217+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:24.217+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:24.217+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:24.218+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:24.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:24.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:24.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:24.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:24.179+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:25.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:24.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:25.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:24.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:25.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:24.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:25.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:24.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:25.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:25.106+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49511 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:25.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:25.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:25.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:25.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:25.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:25.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:25.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:25.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:26.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:25.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:26.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:25.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:26.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:25.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:26.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:25.842+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:26.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:26.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:26.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:26.134+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:26.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:26.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:26.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:26.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:26.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:27.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:26.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:27.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:26.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:27.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:26.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:27.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:26.869+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:27.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:27.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:27.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:27.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:27.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:27.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:27.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:27.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:28.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:27.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:28.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:27.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:28.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:27.918+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:28.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:27.919+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:28.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:28.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:28.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:28.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:28.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:28.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:28.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:28.187+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:29.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:28.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:29.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:28.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:29.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:28.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:29.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:28.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:29.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:29.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:29.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:29.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:29.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:29.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:29.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:29.112+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:29.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:29.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:29.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:29.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:29.200+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:30.298 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:30.113+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49557 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:30.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:30.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:30.123+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:30.124+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:30.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:29.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:30.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:29.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:30.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:29.987+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:30.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:29.988+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:30.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:30.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:30.190+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:30.191+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:31.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:31.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:31.077+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:31.078+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:31.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:31.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:31.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:31.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:31.145+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:31.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:30.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:31.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:30.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:31.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:30.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:31.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:30.965+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:32.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:31.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:32.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:31.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:32.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:31.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:32.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:32.000+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:32.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:32.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:32.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:32.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:32.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:32.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:32.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:32.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:32.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:32.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:32.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:32.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:32.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:32.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:32.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:32.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:33.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:33.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:33.121+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:33.123+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:33.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:33.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:33.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:33.188+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:34.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:34.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:34.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:34.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:34.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:34.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:34.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:34.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:34.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:34.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:34.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:34.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:34.232+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:35.248 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:35.119+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49602 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:35.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:35.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:35.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:35.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:35.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:35.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:35.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:35.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:35.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:35.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:35.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.993+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:35.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:34.994+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:35.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:35.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:35.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:35.246+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:35.964+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:35.964+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:35.964+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:35.964+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:36.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:36.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:36.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:36.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:36.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:36.143+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:36.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:36.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:36.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:36.237+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:36.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:36.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:36.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:37.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:36.917+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:37.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:37.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:37.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:37.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:37.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:37.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:37.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:37.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:37.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:37.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:38.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:38.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:38.175+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:38.176+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:37.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:38.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:37.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:38.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:37.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:38.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:37.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:38.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:38.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:38.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:38.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:39.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:38.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:39.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:38.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:39.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:38.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:39.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:38.933+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:39.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:39.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:39.213+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:39.214+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:39.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:39.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:39.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:39.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:39.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:40.191 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:40.126+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49647 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:40.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:39.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:40.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:39.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:40.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:39.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:40.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:39.930+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:40.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:40.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:40.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:40.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:40.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:40.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:40.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:40.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:40.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:40.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:40.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:41.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:40.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:41.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:40.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:41.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:40.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:41.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:40.928+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:41.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:41.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:41.370+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:41.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:41.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:41.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:41.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:41.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:42.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:41.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:42.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:41.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:42.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:41.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:42.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:41.928+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:42.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:42.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:42.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:42.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:42.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:42.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:42.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:42.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:42.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:42.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:42.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:43.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:42.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:43.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:42.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:43.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:42.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:43.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:42.940+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:43.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:43.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:43.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:43.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:43.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:43.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:43.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:43.210+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:43.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:43.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:43.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:43.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:43.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:43.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:43.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:43.419+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:44.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:43.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:44.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:43.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:44.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:43.913+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:44.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:43.914+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:44.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:44.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:44.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:44.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:44.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:44.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:44.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:44.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:44.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:44.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:44.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:44.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:44.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:44.866+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:45.435 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:45.132+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49692 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:45.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:45.199+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:45.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:45.199+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:45.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:45.199+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:45.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:45.200+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:45.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:45.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:45.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:45.432+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:46.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:45.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:46.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:45.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:46.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:45.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:46.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:45.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:46.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:46.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:46.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:46.204+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:46.205+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:46.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:46.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:46.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:46.393+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:46.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:46.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:46.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:46.790+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:47.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:47.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:47.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:47.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:47.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:47.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:47.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:47.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:47.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:47.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:47.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:47.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:47.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:47.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:47.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:47.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:47.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:47.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:47.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:47.830+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:48.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:48.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:48.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:48.253+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:48.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:48.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:48.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:48.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:48.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:48.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:48.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:48.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:49.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:49.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:49.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:49.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:49.218+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:49.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:49.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:49.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:49.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:49.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:49.384+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:50.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:49.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:50.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:49.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:50.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:49.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:50.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:49.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:50.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:50.138+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49738 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:50.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:50.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:50.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:50.268+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:50.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:50.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:50.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:50.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:50.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:50.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:50.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:50.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:50.869+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:51.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:51.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:51.267+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:51.268+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:51.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:51.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:51.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:51.364+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:51.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:51.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:51.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:51.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:52.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:52.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:52.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:52.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:52.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:52.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:52.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:52.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:52.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:52.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:52.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:52.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:52.830+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:53.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:53.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:53.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:53.250+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:53.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:53.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:53.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:53.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:53.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:53.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:53.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:53.794+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:54.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:54.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:54.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:54.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:54.269+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:54.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:54.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:54.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:54.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:54.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:54.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:54.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:54.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:55.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:54.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:55.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:54.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:55.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:54.773+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:55.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:54.774+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:55.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:54:55.144+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49783 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:54:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:55.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:55.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:55.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:55.340+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:55.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:55.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:55.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:55.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:55.760+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:55.760+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:55.760+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:55.761+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:56.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:56.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:56.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:56.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:56.368+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:56.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:56.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:56.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:56.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:56.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:56.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:56.717+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:56.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:56.717+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:57.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:57.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:57.363+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:57.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:57.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:57.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:57.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:57.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:57.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:57.213+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:57.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:57.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:57.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:57.749+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:58.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:58.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:58.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:58.353+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:58.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:58.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:58.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:58.201+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:58.701+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:58.701+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:58.701+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:58.701+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:59.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:59.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:59.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:54:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:54:59.379+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:59.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:59.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:59.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:59.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:54:59.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:54:59.196+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:54:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:59.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:59.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:59.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:54:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:54:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:54:59.685+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:00.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:00.151+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49828 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:00.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:00.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:00.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:00.377+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:00.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:00.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:00.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:00.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:00.247+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:00.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:00.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:00.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:00.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:00.644+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:01.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:01.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:01.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:01.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:01.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:01.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:01.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:01.198+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:01.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:01.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:01.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:01.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:01.681+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:02.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:02.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:02.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:02.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:02.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:02.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:02.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:02.170+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:02.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:02.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:02.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:02.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:02.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:02.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:02.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:02.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:02.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:03.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:03.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:03.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:03.185+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:03.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:03.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:03.691+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:03.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:03.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:03.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:03.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:03.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:04.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:04.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:04.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:04.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:04.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:04.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:04.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:04.164+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:04.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:04.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:04.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:04.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:04.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:04.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:04.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:04.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:04.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:04.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:04.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:04.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:05.439 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:05.157+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49873 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:05.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:05.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:05.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:05.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:05.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:05.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:05.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:05.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:05.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:05.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:05.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:05.437+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:05.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:05.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:05.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:05.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:05.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:05.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:05.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:05.677+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:06.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:06.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:06.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:06.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:06.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:06.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:06.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:06.159+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:06.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:06.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:06.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:06.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:06.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:06.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:06.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:06.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:07.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:07.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:07.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:07.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:07.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:07.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:07.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:07.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:07.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:07.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:07.671+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:07.672+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:07.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:07.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:07.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:07.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:08.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:08.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:08.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:08.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:08.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:08.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:08.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:08.137+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:08.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:08.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:08.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:08.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:08.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:08.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:08.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:08.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:09.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:09.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:09.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:09.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:09.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:09.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:09.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:09.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:09.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:09.726+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:09.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:09.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:09.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:09.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:09.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:09.492+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:10.473 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:10.164+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49919 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:10.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:10.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:10.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:10.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:10.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:10.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:10.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:10.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:10.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:10.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:10.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:10.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:11.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:10.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:11.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:10.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:11.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:10.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:11.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:10.757+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:11.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:11.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:11.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:11.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:11.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:11.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:11.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:11.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:11.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:11.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:11.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:11.775+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:11.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:11.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:11.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:11.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:11.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:11.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:11.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:11.536+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:12.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:12.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:12.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:12.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:12.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:12.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:12.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:12.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:12.827 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:12.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:12.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:12.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:13.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:12.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:13.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:12.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:13.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:12.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:13.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:12.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:13.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:13.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:13.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:13.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:13.157+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:13.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:13.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:13.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:13.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:13.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:13.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:13.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:13.576+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:14.170 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:13.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:14.170 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:13.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:14.170 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:13.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:14.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:13.830+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:14.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:14.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:14.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:14.167+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:14.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:14.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:14.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:14.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:14.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:14.595+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:14.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:14.596+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:15.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:14.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:15.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:14.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:15.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:14.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:15.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:14.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:15.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:15.170+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49964 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:15.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:15.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:15.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:15.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:15.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:15.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:15.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:15.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:15.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:15.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:15.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:15.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:16.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:15.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:16.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:15.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:16.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:15.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:16.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:15.871+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:16.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:16.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:16.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:16.175+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:16.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:16.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:16.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:16.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:16.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:16.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:16.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:16.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:17.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:16.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:17.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:16.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:17.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:16.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:17.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:16.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:17.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:17.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:17.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:17.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:17.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:17.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:17.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:17.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:17.850 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:17.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:17.850 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:17.557+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:17.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:17.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:17.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:17.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:18.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:18.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:18.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:18.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:18.219+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:18.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:18.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:18.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:18.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:18.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:18.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:18.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:18.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:18.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:18.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:18.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:18.822+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:19.560 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:19.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:19.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:19.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:19.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:19.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:19.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:19.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:19.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:19.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:19.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:19.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:19.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:20.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:19.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:20.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:19.847+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:20.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:19.848+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:20.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:19.849+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:20.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:20.177+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50009 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:20.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:20.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:20.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:20.259+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:20.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:20.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:20.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:20.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:20.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:20.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:20.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:20.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:20.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:20.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:21.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:21.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:21.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:21.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:21.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:21.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:21.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:21.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:21.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:21.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:21.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:21.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:21.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:21.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:21.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:21.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:21.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:21.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:21.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:21.930+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:22.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:22.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:22.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:22.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:22.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:22.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:22.598 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:22.248+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:22.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:22.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:22.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:22.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:22.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:22.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:22.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:22.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:22.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:22.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:22.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:22.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:23.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:23.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:23.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:23.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:23.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:23.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:23.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:23.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:23.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:23.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:23.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:23.571+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:24.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:23.919+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:23.919+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:23.919+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:23.920+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:24.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:24.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:24.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:24.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:24.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:24.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:24.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:24.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:24.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:24.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:24.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:24.610+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:25.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:25.183+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50054 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:24.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:24.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:24.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:25.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:24.969+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:25.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:25.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:25.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:25.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:25.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:25.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:25.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:25.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:25.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:25.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:25.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:25.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:25.608+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:26.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:26.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:26.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:26.009+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:26.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:26.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:26.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:26.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:26.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:26.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:26.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:26.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:27.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:27.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:27.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:27.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:27.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:27.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:27.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:27.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:27.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:27.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:27.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:27.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:27.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:27.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:27.570 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:27.300+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:27.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:27.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:27.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:27.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:28.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:28.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:28.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:28.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:28.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:28.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:28.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:28.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:28.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:28.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:28.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:28.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:29.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:29.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:29.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:29.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:28.985+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:29.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:29.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:29.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:29.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:29.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:29.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:29.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:29.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:29.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:29.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:29.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:29.594+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:30.302 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:30.190+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50100 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:30.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:29.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:30.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:29.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:30.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:29.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:30.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:29.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:30.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:30.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:30.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:30.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:30.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:30.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:30.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:30.347+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:30.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:30.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:30.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:30.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:30.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:30.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:30.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:30.592+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:30.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:30.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:30.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:30.935+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:31.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:31.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:31.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:31.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:31.570+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:31.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:31.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:31.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:31.353+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:31.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:31.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:31.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:31.969+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:32.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:32.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:32.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:32.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:32.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:32.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:32.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:32.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:33.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:32.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:33.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:32.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:32.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:32.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:33.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:33.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:33.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:33.540+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:33.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:33.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:33.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:33.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:34.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:34.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:34.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:34.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:34.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:34.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:34.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:34.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:34.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:34.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:34.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:34.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:34.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:34.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:34.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:34.399+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:35.323 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:35.196+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50145 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:35.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:35.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:35.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:35.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:35.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:35.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:35.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:35.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:35.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:35.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:35.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:35.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:35.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:35.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:35.356+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:35.357+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:36.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:36.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:36.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:36.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:36.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:36.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:36.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:36.028+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:36.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:36.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:36.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:36.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:36.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:36.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:36.352+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:36.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:37.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:37.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:37.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:37.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:37.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:37.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:37.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:37.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:37.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:37.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:37.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:37.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:37.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:37.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:37.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:37.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:37.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:38.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:38.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:38.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:38.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:38.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:38.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:38.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:38.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:38.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:38.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:38.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:38.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:38.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:38.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:38.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:38.362+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:39.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:39.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:39.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:39.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:39.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:39.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:39.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:39.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:39.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:39.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:39.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:39.448+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:39.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:39.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:39.359+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:39.360+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:40.302 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:40.202+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50190 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:40.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:40.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:40.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:40.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:40.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:40.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:40.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:40.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:40.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:40.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:40.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:40.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:40.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:40.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:40.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:40.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:40.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:41.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:41.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:41.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:41.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:41.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:41.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:41.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:41.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:41.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:41.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:41.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:41.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:41.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:41.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:41.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:41.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:42.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:42.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:42.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:42.036+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:42.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:42.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:42.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:42.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:42.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:42.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:42.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:42.448+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:43.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:43.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:43.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:43.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:43.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:43.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:43.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:43.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:43.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:43.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:43.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:43.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:43.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:43.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:43.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:43.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:44.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:44.013+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:44.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:44.013+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:44.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:44.013+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:44.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:44.014+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:44.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:44.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:44.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:44.427+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:44.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:44.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:44.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:44.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:45.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:45.208+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50235 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:45.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:45.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:45.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:45.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:45.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:45.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:45.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:45.003+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:45.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:45.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:45.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:45.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:45.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:45.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:45.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:45.446+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:46.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:46.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:46.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:46.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:46.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:46.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:46.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:46.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:46.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:46.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:46.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:46.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:46.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:46.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:46.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:46.493+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:47.336 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:47.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:47.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:47.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:47.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:47.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:47.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:47.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:47.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:47.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:47.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:47.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:47.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:47.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:47.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:47.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:47.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:47.502+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:48.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:48.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:48.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:48.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:48.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:48.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:48.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:48.066+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:48.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:48.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:48.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:48.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:48.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:48.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:48.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:48.498+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:49.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:49.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:49.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:49.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:49.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:49.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:49.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:49.062+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:49.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:49.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:49.393+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:49.394+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:49.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:49.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:49.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:49.453+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:50.341 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:50.214+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50281 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:50.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:50.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:50.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:50.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:50.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:50.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:50.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:50.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:50.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:50.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:50.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:50.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:50.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:50.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:50.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:50.483+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:51.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:51.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:51.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:51.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:51.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:51.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:51.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:51.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:51.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:51.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:51.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:51.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:51.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:51.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:51.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:51.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:51.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:52.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:52.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:52.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:52.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:52.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:52.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:52.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:52.492+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:52.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:52.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:52.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:52.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:52.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:53.088+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:53.088+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:53.088+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:53.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:53.089+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:53.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:53.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:53.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:53.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:53.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:53.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:53.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:53.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:54.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:54.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:54.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:54.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:54.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:54.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:54.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:54.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:54.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:54.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:54.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:54.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:54.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:54.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:54.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:54.479+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:55.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:55:55.220+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50326 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:55:55.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:55.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:55.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:55.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:55.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:55.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:55.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:55.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:55.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:55.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:55.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:55.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:55.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:55.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:55.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:55.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:56.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:56.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:56.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:56.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:56.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:56.001+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:56.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:56.002+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:56.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:56.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:56.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:56.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:56.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:56.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:56.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:56.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:56.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:57.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:57.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:57.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:57.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:57.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:57.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:57.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:57.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:57.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:57.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:57.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:57.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:57.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:57.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:57.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:57.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:58.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:58.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:58.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:58.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:58.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:58.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:58.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:58.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:58.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:58.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:58.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:58.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:58.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:58.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:58.548+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:58.549+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:59.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:59.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:59.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:59.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:59.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:59.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:55:59.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:55:59.073+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:55:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:59.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:59.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:59.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:55:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:55:59.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:59.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:59.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:59.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:55:59.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:55:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:55:59.568+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:00.352 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:00.226+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50371 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:00.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:00.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:00.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:00.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:00.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:00.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:00.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:00.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:00.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:00.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:00.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:00.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:00.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:00.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:00.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:00.398+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:00.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:00.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:00.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:00.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:01.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:01.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:01.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:01.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.031+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:01.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:01.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:01.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:01.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:01.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:01.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:01.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:01.602+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:01.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:02.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:02.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:02.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:02.572+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:02.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:02.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:02.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:02.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:02.452+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:03.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:03.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:03.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.011+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:03.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:03.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:03.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:03.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:03.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:03.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:03.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:03.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:03.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:03.605+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:04.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:03.971+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:04.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:04.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:04.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:04.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:04.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:04.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:04.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:04.585+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:04.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:04.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:04.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:04.959+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:05.565 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:05.232+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50417 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:05.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:05.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:05.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:05.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:05.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:05.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:05.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:05.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:05.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:05.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:05.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:05.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:05.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:05.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:05.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:05.562+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:05.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:05.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:05.963+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:05.964+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:06.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:06.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:06.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:06.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:06.543+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:06.543+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:06.543+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:06.543+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:06.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:06.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:06.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:06.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:07.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:07.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:07.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:07.337+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:07.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:07.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:07.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:07.538+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:07.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:07.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:07.938+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:07.939+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:08.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:08.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:08.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:08.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:08.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:08.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:08.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:08.492+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:09.336 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:08.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:09.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:08.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:09.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:08.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:09.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:08.982+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:09.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:09.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:09.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:09.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:09.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:09.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:09.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:09.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:09.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:09.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:09.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:09.941+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:10.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:10.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:10.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:10.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:10.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:10.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:10.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:10.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:10.689 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:10.238+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50461 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:10.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:10.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:10.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:10.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:10.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:10.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:10.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:10.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:10.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:10.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:10.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:10.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:10.912+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:11.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:11.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:11.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:11.274+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:11.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:11.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:11.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:11.440+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:11.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:11.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:11.954+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:11.955+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:12.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:12.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:12.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:12.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:12.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:12.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:12.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:12.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:12.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:13.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:12.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:13.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:12.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:13.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:12.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:13.305 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:12.997+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:13.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:13.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:13.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:13.302+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:13.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:13.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:13.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:13.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:13.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:13.460+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:14.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:14.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:14.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:14.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:14.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:14.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:14.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:14.015+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:14.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:14.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:14.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:14.338+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:14.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:14.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:14.423+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:15.366 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:15.245+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50499 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:15.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:15.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:15.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:15.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:15.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:15.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:15.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:15.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:15.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:15.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:15.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:15.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:15.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:15.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:15.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:15.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:15.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:15.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:16.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:16.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:16.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:16.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:16.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:16.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:16.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:16.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:16.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:16.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:16.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:16.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:16.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:16.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:16.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:16.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:16.476+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:17.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:17.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:17.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:17.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.006+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:17.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:17.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:17.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:17.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:17.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:17.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:17.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:17.342+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:17.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:17.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:17.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:17.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:17.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:17.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:17.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:17.454+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:18.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:18.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:18.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:18.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:17.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:18.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:18.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:18.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:18.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:18.391+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:18.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:18.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:18.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:18.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:18.454+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:19.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:18.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:19.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:18.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:19.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:18.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:19.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:18.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:19.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:19.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:19.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:19.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:19.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:19.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:19.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:19.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:19.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:19.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:19.414+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:20.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:20.251+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50543 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:20.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:20.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:20.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:20.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:20.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:20.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:20.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:20.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:20.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:20.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:20.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:20.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:20.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:20.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:20.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:20.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:20.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:20.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:20.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:20.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:21.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:21.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:21.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:21.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:21.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:21.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:21.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:21.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:21.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:21.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:21.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:21.451+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:21.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:21.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:21.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:21.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:21.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:21.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:22.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:22.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:22.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:22.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:22.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:22.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:22.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:22.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:22.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:22.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:22.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:22.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:22.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:22.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:23.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:23.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:23.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:23.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:23.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:23.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:23.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:23.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:23.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:23.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:23.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:23.353+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:23.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:23.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:23.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:23.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:23.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:23.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:23.484+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:24.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:24.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:24.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:24.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:24.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:24.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:24.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:24.324+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:24.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:24.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:24.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:24.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:24.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:24.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:24.504+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:25.261 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:25.261 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:25.261 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.994+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:25.261 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:24.994+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:25.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:25.257+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50589 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:25.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:25.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:25.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:25.337+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:25.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:25.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:25.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:25.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:25.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:25.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:25.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:25.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:25.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:26.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:26.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:26.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:26.540+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:26.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:26.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:26.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:26.296+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:27.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:27.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:27.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:27.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:27.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:27.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:27.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:27.010+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:27.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:27.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:27.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:27.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:27.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:27.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:27.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:27.497+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:28.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:28.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:28.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:28.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:28.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:28.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:28.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:28.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:28.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:28.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:28.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:28.493+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:29.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:29.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:29.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:29.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:29.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:29.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:29.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:29.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:29.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:29.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:30.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:30.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:30.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:30.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:30.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:30.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:30.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:30.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:30.389 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:30.264+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50634 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:30.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:30.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:30.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:30.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:30.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:30.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:30.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:30.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:30.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:30.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:30.455+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:30.456+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:31.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:31.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:31.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:31.050+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:31.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:31.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:31.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:31.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:31.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:31.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:31.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:31.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:31.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:31.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:31.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:31.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:31.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:31.475+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:32.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:32.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:32.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:32.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:32.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:32.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:32.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:32.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:32.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:32.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:32.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:32.296+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:32.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:32.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:32.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:32.480+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:33.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:33.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:33.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:33.082+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:33.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:33.083+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:33.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:33.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:33.280+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:33.281+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:33.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:33.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:33.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:33.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:34.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:34.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:34.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:34.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:34.298 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:34.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:34.298 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:34.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:34.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:34.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:34.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:34.295+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:34.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:34.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:34.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:34.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:35.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:35.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:35.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:35.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:35.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:35.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:35.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:35.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:35.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:35.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:35.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:35.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:35.396 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:35.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:35.396 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:35.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:35.396 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:35.271+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50679 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:35.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:35.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:35.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:35.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:36.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:36.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:36.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:36.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:36.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:36.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:36.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:36.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:36.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:36.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:36.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:36.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:36.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:37.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:37.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:37.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:37.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:37.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:37.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:37.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:37.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:37.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:37.258+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:37.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:37.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:37.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:37.431+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:38.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:38.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:38.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:38.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:38.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:38.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:38.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:38.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:38.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:38.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:38.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:38.270+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:38.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:38.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:38.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:38.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:38.392+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:39.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:39.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:39.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.033+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:39.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.034+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:39.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:39.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:39.318+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:39.319+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:39.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:39.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:39.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:39.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:39.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:40.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:40.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:40.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:40.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:39.990+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:40.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:40.277+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50724 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:40.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:40.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:40.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:40.348+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:40.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:40.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:40.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:40.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:40.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:40.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:40.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:41.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:40.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:41.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:40.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:41.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:40.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:41.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:40.976+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:41.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:41.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:41.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:41.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:41.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:41.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:41.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:41.361+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:41.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:41.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:41.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:42.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:41.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:42.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:42.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:42.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:42.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:42.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:42.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:42.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:42.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:42.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:42.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:42.895+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:42.897+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:43.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:43.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:43.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:43.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:43.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:43.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:43.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:43.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:43.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:43.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:43.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:44.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:43.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:44.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:43.857+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:44.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:43.858+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:44.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:44.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:44.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:44.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:44.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:44.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:44.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:44.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:44.340+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:44.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:44.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:44.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:45.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:44.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:45.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:45.283+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50770 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:45.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:45.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:45.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:45.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:45.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:45.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:45.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:45.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:45.329+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:45.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:45.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:45.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:45.853+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:46.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:46.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:46.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:46.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:46.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:46.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:46.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:46.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:46.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:46.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:46.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:46.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:47.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:46.807+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:47.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:47.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:47.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:47.375+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:47.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:47.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:47.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:47.408+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:47.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:47.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:47.758+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:47.758+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:47.758+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:47.759+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:48.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:48.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:48.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:48.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:48.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:48.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:48.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:48.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:48.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:48.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:48.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:48.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:49.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:49.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:49.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:49.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:49.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:49.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:49.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:49.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:49.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:49.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:49.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:49.789+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:49.790+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:50.624 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:50.291+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50815 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:50.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:50.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:50.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:50.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:50.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:50.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:50.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:50.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:50.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:50.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:50.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:50.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:50.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:50.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:50.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:50.383+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:50.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:50.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:50.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:50.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:50.779+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:51.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:51.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:51.475+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:51.476+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:51.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:51.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:51.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:51.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:51.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:51.395+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:51.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:51.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:51.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:51.811+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:52.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:52.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:52.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:52.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:52.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:52.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:52.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:52.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:52.374+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:52.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:52.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:52.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:52.763+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:53.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:53.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:53.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:53.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:53.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:53.375+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:53.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:53.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:53.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:53.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:53.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:53.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:53.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:53.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:53.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:53.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:53.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:53.758+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:54.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:54.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:54.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:54.502+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:54.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:54.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:54.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:54.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:54.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:54.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:54.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:54.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:54.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:54.782+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:55.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:56:55.298+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50860 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:56:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:55.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:55.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:55.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:55.510+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:55.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:55.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:55.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:55.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:55.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:55.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:55.383+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:55.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:55.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:55.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:55.754+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:56.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:56.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:56.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:56.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:56.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:56.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:56.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:56.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:56.511+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:56.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:56.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:56.776+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:56.776+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:57.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:57.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:57.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:57.334+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:57.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:57.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:57.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:57.527+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:57.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:57.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:57.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:57.794+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:58.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:58.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:58.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:58.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:58.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:58.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:58.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:58.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:58.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:58.536+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:58.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:58.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:58.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:56:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:58.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:56:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:59.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:59.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:59.352+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:56:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:56:59.353+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:59.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:59.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:59.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:56:59.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:56:59.529+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:59.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:59.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:59.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:56:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:56:59.856+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:00.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:00.305+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50905 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:00.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:00.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:00.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:00.341+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:00.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:00.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:00.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:00.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:00.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:00.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:00.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:00.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:00.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:00.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:01.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:01.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:01.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:01.330+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:01.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:01.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:01.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:01.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:01.518+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:01.804+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:01.804+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:01.804+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:01.806+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:02.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:02.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:02.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:02.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:02.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:02.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:02.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:02.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:02.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:02.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:02.506+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:02.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:02.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:02.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:02.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:03.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:03.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:03.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:03.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:03.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:03.468+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:03.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:03.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:03.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:03.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:03.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:03.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:03.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:03.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:04.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:03.822+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:03.822+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:03.822+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:03.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:04.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:04.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:04.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:04.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:04.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:04.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:04.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:04.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:04.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:04.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:04.842+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:04.843+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:05.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:05.312+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50950 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:05.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:05.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:05.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:05.466+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:05.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:05.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:05.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:05.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:05.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:05.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:05.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:05.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:05.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:05.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:05.862+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:06.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:06.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:06.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:06.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:06.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:06.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:06.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:06.408+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:06.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:06.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:06.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:06.887+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:07.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:07.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:07.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:07.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:07.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:07.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:07.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:07.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:07.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:07.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:07.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:07.862+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:08.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:08.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:08.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:08.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:08.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:08.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:08.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:08.356+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:08.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:08.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:08.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:08.842+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:09.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:09.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:09.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:09.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:09.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:09.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:09.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:09.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:09.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:09.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:09.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:09.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:10.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:10.318+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 50997 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:10.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:10.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:10.447+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:10.448+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:10.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:10.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:10.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:10.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:10.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:10.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:10.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:10.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:10.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:10.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:10.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:10.880+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:11.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:11.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:11.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:11.240+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:11.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:11.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:11.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:11.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:11.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:11.466+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:12.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:11.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:12.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:11.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:12.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:11.830+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:12.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:11.831+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:12.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:12.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:12.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:12.196+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:12.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:12.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:12.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:12.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:12.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:12.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:12.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:12.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:12.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:12.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:12.815+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:12.816+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:13.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:13.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:13.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:13.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:13.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:13.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:13.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:13.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:13.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:13.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:14.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:13.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:14.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:13.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:14.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:13.863+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:14.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:14.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:14.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:14.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:14.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:14.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:14.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:14.240+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:14.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:14.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:14.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:14.511+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:15.200 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:14.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:15.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:14.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:15.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:14.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:15.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:14.870+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:15.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:15.324+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51041 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:15.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:15.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:15.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:15.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:15.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:15.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:15.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:15.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:15.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:15.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:15.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:15.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:15.909+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:15.909+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:15.909+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:16.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:15.910+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:16.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:16.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:16.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:16.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:16.217+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:16.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:16.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:16.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:16.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:16.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:16.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:16.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:16.514+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:17.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:16.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:17.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:16.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:17.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:16.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:17.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:16.870+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:17.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:17.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:17.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:17.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:17.889 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:17.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:17.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:17.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:17.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:17.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:17.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:17.540+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:17.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:17.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:17.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:17.887+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:18.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:18.240+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:18.240+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:18.240+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:18.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:18.241+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:18.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:18.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:18.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:18.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:18.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:18.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:18.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:18.577+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:19.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:18.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:19.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:18.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:19.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:18.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:19.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:18.862+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:19.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:19.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:19.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:19.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:19.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:19.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:19.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:19.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:19.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:19.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:19.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:19.574+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:19.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:19.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:19.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:19.908+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:20.560 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:20.331+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51087 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:20.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:20.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:20.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:20.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:20.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:20.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:20.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:20.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:20.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:20.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:20.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:20.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:20.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:20.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:20.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:20.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:21.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:20.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:21.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:20.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:21.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:20.931+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:21.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:20.932+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:21.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:21.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:21.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:21.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:21.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:21.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:21.853 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:21.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:21.853 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:21.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:21.853 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:21.593+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:22.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:21.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:22.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:21.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:22.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:21.922+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:22.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:21.923+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:22.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:22.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:22.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:22.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:22.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:22.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:22.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:22.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:22.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:22.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:22.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:22.604+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:23.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:22.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:23.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:22.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:23.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:22.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:23.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:22.905+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:23.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:23.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:23.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:23.209+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:23.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:23.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:23.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:23.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:23.876 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:23.610+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:23.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:23.611+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:24.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:23.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:24.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:23.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:24.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:23.873+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:24.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:23.874+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:24.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:24.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:24.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:24.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:24.199+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:24.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:24.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:24.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:24.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:24.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:24.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:24.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:24.581+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:25.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:24.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:25.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:24.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:25.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:24.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:25.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:24.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:25.459 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:25.337+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51132 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:25.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:25.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:25.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:25.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:25.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:25.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:25.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:25.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:25.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:25.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:25.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:25.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:26.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:25.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:26.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:25.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:26.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:25.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:26.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:25.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:26.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:26.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:26.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:26.197+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:26.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:26.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:26.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:26.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:26.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:26.537+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:26.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:26.538+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:27.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:26.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:27.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:26.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:27.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:26.869+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:27.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:26.870+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:27.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:27.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:27.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:27.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:27.178+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:27.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:27.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:27.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:27.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:27.860 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:27.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:27.860 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:27.523+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:27.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:27.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:27.855+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:28.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:27.856+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:28.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:28.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:28.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:28.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:28.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:28.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:28.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:28.223+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:28.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:28.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:28.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:28.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:28.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:28.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:28.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:28.513+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:28.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:28.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:28.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:29.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:28.858+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:29.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:29.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:29.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:29.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:29.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:29.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:29.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:29.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:29.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:29.273+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:30.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:29.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:30.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:29.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:30.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:29.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:30.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:29.880+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:30.560 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:30.344+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51178 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:30.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:30.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:30.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:30.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:30.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:30.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:30.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:30.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:30.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:30.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:30.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:30.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:30.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:30.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:30.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:30.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:30.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:30.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:30.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:30.838+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:31.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:31.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:31.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:31.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:31.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:31.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:31.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:31.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:31.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:31.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:31.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:31.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:32.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:32.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:32.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:32.275+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:32.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:32.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:32.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:32.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:32.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:33.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:32.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:33.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:32.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:33.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:32.813+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:33.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:33.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:33.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:33.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:33.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:33.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:33.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:33.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:33.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:33.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:33.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:33.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:33.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:33.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:33.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:33.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:33.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:33.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:33.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:33.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:33.853+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:34.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:34.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:34.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:34.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:34.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:34.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:34.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:34.237+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:34.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:34.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:34.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:34.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:34.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:34.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:34.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:34.511+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:35.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:34.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:34.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:34.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:34.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:35.517 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:35.350+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51222 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:35.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:35.240+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:35.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:35.240+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:35.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:35.240+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:35.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:35.241+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:35.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:35.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:35.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:35.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:35.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:35.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:35.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:35.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:36.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:35.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:36.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:35.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:35.881+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:36.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:35.882+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:36.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:36.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:36.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:36.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:36.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:36.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:36.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:36.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:36.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:36.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:36.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:36.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:36.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:36.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:36.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:36.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:36.851+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:36.851+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:36.851+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:36.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:36.852+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:37.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:37.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:37.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:37.227+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:37.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:37.227+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:37.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:37.228+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:37.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:37.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:37.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:37.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:37.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:37.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:37.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:37.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:37.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:37.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:37.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:38.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:37.900+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:38.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:38.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:38.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:38.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:38.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:38.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:38.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:38.240+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:38.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:38.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:38.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:38.879+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:39.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:39.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:39.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:39.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:39.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:39.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:39.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:39.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:39.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:39.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:39.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:39.223+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:39.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:39.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:39.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:39.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:40.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:39.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:40.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:39.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:40.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:39.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:40.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:39.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:40.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:40.357+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51268 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:40.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:40.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:40.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:40.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:40.176+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:40.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:40.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:40.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:40.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:40.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:40.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:40.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:40.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:41.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:40.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:41.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:40.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:41.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:40.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:41.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:40.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:41.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:41.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:41.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:41.168+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:41.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:41.169+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:41.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:41.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:41.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:41.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:42.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:41.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:42.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:41.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:42.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:41.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:42.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:41.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:42.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:42.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:42.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:42.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:42.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:42.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:42.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:42.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:42.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:43.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:42.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:43.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:42.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:43.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:42.879+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:43.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:42.880+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:43.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:43.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:43.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:43.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:43.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:43.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:43.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:43.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:43.196+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:44.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:43.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:44.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:43.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:44.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:43.910+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:44.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:43.910+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:44.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:44.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:44.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:44.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:44.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:44.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:44.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:44.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:44.216+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:45.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:44.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:45.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:44.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:45.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:44.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:45.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:44.937+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:45.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:45.363+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51313 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:45.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:45.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:45.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:45.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:45.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:45.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:45.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:45.206+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:45.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:45.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:45.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:45.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:46.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:46.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:46.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:46.228+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:46.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:46.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:46.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:46.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:46.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:47.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:46.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:47.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:46.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:47.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:46.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:47.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:46.954+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:47.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:47.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:47.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:47.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:47.400+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:47.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:47.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:47.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:47.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:47.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:47.211+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:48.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:47.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:48.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:47.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:48.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:47.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:48.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:47.909+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:48.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:48.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:48.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:48.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:48.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:48.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:48.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:48.198+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:48.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:48.199+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:48.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:48.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:48.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:48.953+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:49.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:49.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:49.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:49.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:49.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:49.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:49.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:49.388+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:49.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:49.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:49.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:49.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:50.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:50.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:50.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:50.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:50.493 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:50.370+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51359 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:50.493 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:50.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:50.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:50.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:50.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:50.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:50.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:50.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:50.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:51.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:51.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:51.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:51.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:51.180+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:51.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:51.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:51.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:51.331+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:52.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:51.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:52.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:51.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:52.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:51.966+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:52.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:51.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:52.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:52.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:52.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:52.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:52.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:52.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:52.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:52.213+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:53.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:53.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:53.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:53.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:53.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:53.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:53.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:53.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:53.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:53.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:53.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:53.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:53.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:53.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:53.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:53.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:53.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:53.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:53.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:53.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:54.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:54.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:54.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:54.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:54.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:54.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:54.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:54.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:54.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:54.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:54.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:54.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:54.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:54.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:54.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:54.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:54.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:54.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:54.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:54.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:55.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:55.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:55.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:55.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:55.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:55.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:55.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:55.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:55.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:55.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:55.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:55.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:55.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:55.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:55.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:55.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:55.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:57:55.376+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51403 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:57:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:55.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:55.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:55.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:55.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:56.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:56.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:56.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:56.083+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:56.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:56.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:56.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:56.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:56.312+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:56.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:56.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:56.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:56.315+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:57.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:57.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:57.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:57.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:57.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:57.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:57.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:57.048+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:57.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:57.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:57.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:57.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:57.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:57.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:57.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:57.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:57.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:58.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:58.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:58.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:58.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:58.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:58.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:58.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:58.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:58.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:58.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:58.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:58.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:58.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:58.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:58.334 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:58.282+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:58.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:58.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:58.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:58.330+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:59.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:59.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:59.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:57:59.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:59.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:59.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:59.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:57:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:57:59.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:57:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:59.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:59.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:59.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:57:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:57:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:57:59.316+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:00.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:00.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:00.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:00.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:00.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:00.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:00.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:00.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:00.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:00.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:00.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:00.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:00.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:00.303+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:00.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:00.304+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:00.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:00.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:00.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:00.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:00.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:00.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:00.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:00.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:00.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:00.383+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51449 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:01.290+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:01.290+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:01.290+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:01.291+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:01.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:01.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:01.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:01.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:01.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:01.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:01.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:01.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:01.355+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:02.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:02.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:02.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:02.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:02.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:02.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:02.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:02.262+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:02.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:02.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:02.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:02.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:02.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:02.388+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:03.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:03.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:03.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:03.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:03.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:03.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:03.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:03.057+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:03.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:03.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:03.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:03.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:03.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:03.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:03.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:03.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:03.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:03.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:03.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:03.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:03.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:04.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:04.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:04.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:04.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:04.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:04.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:04.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:04.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:04.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:04.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:04.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:04.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:04.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:04.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:04.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:04.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:05.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:05.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:05.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:05.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:05.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:05.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:05.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:05.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:05.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:05.389+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51494 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:05.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:05.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:05.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:05.326+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:05.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:05.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:05.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:05.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:06.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:06.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:06.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:06.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:06.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:06.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:06.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:06.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:06.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:06.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:06.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:06.359+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:06.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:06.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:06.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:06.337+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:07.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:07.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:07.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:07.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:07.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:07.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:07.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:07.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:07.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:07.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:07.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:07.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:07.341+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:08.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:08.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:08.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:08.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:08.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:08.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:08.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:08.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:08.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:08.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:08.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:08.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:08.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:08.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:08.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:08.269+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:08.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:08.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:08.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:08.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:09.305 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:09.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:09.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:09.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:09.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:09.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:09.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:09.031+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:09.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:09.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:09.382+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:09.383+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:09.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:09.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:09.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:09.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:10.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:10.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:10.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:10.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:10.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:10.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:10.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:10.071+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:10.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:10.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:10.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:10.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:10.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:10.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:10.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:10.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:10.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:10.397+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51540 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:10.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:10.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:10.334+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:10.335+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:11.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:11.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:11.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:11.302+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:11.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:11.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:11.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:11.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:11.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:11.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:11.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:11.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:11.316+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:12.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:12.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:12.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:12.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:12.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:12.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:12.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:12.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:12.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:12.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:12.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:12.345+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:13.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:13.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:13.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:13.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:13.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:13.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:13.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:13.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:13.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:13.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:13.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:13.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:13.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:13.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:13.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:13.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:13.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:13.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:13.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:13.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:14.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:14.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:14.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:14.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:14.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:14.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:14.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:14.064+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:14.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:14.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:14.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:14.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:14.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:14.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:14.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:14.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:14.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:15.305 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:15.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:15.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:15.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:15.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:15.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:15.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:15.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:15.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:15.403+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51584 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:15.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:15.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:15.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:15.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:15.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:15.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:15.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:15.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:15.388+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:16.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:16.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:16.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:16.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:16.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:16.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:16.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:16.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:16.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:16.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:16.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:16.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:16.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:16.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:16.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:16.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:16.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:16.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:16.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:16.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:16.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:16.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:16.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:16.340+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:17.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:17.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:17.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:17.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:17.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:17.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:17.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:17.279+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:17.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:17.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:17.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:17.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:17.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:17.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:17.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:17.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:17.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:17.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:17.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:17.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:17.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:17.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:17.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:17.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:18.325 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:18.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:18.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:18.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:18.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:18.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:18.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:18.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:18.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:18.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:18.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:18.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:18.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:18.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:18.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:18.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:18.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:18.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:18.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:18.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:19.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:19.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:19.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:19.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:19.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:19.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:19.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:19.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:19.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:19.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:19.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:19.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:19.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:19.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:19.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:19.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:19.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:19.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:19.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:20.309 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:20.309 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:20.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:20.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:20.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:20.409+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51630 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:20.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:20.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:20.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:20.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:20.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:20.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:20.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:20.337+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:21.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:21.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:21.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:21.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:20.985+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:21.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:21.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:21.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:21.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:21.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:21.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:21.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:21.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:21.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:21.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:21.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:21.973+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:22.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:22.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:22.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:22.416+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:22.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:22.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:22.342+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:22.343+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:22.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:22.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:22.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:22.977+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:23.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:23.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:23.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:23.375+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:23.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:23.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:23.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:23.368+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:24.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:24.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:24.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:24.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:24.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:24.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:24.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:24.004+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:24.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:24.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:24.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:24.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:24.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:24.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:24.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:24.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:24.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:25.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:25.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:25.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:25.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.003+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:25.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:25.416+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51675 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:25.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:25.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:25.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:25.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:25.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:25.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:25.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:25.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:25.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:25.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.991+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:25.992+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:26.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:26.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:26.327+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:26.328+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:26.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:26.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:26.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:26.431+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:27.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:27.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:27.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:27.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:27.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:27.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:27.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:27.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:27.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:27.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:27.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:27.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:27.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:27.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:27.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:27.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:27.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:27.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:27.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:27.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:27.439+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:28.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:28.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:28.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:28.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:28.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:28.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:28.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:28.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:28.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:28.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:28.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:28.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:28.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:28.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:28.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:28.338+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:29.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:29.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:29.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:29.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:29.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:29.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:29.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:29.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:29.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:29.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:29.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:29.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:29.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:29.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:29.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:29.307+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:29.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:29.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:29.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:29.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:30.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:30.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:30.310 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:30.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:30.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:30.422+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51722 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:30.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:30.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:30.506+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:30.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:30.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:30.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:30.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:30.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:30.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:30.970+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:31.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:31.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:31.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:31.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:31.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:31.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:31.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:31.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:32.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:32.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:32.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:32.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:32.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:32.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:32.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:32.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:32.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:32.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:32.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:32.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:33.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:33.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:33.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:33.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:33.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:33.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:33.323 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:33.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:33.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:33.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:33.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:33.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:33.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:33.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:33.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:33.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:34.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:34.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:34.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:34.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:34.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:34.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:34.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:34.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:34.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:34.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:34.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:34.430+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:34.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:34.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:34.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:34.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:35.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:35.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:35.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:35.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:35.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:35.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:35.428+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51766 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:35.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:35.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:35.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:35.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:35.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:35.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:35.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:35.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:35.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:35.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:36.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:36.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:36.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:36.076+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:36.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:36.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:36.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:36.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:36.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:36.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:36.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:36.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:36.419+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:37.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:37.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:37.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:37.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:37.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:37.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:37.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:37.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:37.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:37.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:37.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:37.452+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:37.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:37.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:37.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:37.416+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:38.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:38.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:38.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:38.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:38.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:38.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:38.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:38.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:38.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:38.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:38.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:38.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:38.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:38.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:38.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:38.393+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:39.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:39.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:39.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:39.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:39.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:39.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:39.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:39.082+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:39.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:39.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:39.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:39.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:39.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:39.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:39.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:39.441+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:40.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:40.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:40.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:40.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:40.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:40.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:40.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:40.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:40.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:40.436+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51812 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:40.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:40.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:40.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:40.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:40.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:40.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:40.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:40.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:41.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:41.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:41.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:41.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:41.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:41.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:41.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:41.068+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:41.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:41.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:41.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:41.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:41.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:41.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:41.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:41.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:42.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:42.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:42.051+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:42.052+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:42.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:42.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:42.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:42.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:42.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:42.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:42.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:42.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:42.434+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:43.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:43.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:43.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:43.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:43.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:43.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:43.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:43.092+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:43.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:43.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:43.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:43.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:43.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:43.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:43.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:43.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:44.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:44.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:44.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:44.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:44.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:44.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:44.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:44.048+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:44.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:44.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:44.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:44.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:44.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:44.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:44.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:44.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:45.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:45.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:45.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:45.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:45.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:45.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:45.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:45.020+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:45.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:45.443+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51857 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:45.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:45.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:45.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:45.499+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:45.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:45.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:45.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:45.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:45.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:45.425+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:46.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:46.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.004+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:46.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:46.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:46.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:46.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:46.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:46.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:46.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:46.546+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:47.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:47.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:47.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.986+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:47.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:46.987+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:47.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:47.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:47.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:47.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:47.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:47.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:47.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:47.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:48.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:48.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:48.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:48.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:48.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:48.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:48.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:48.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:48.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:48.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:48.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:48.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:48.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:48.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:48.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:48.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:49.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:49.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:49.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:49.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:49.042+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:49.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:49.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:49.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:49.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:49.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:49.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:49.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:49.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:49.605+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:50.312 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:50.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:50.312 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:50.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:50.312 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:50.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:50.312 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:50.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:50.572 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:50.448+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51903 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:50.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:50.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:50.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:50.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:50.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:50.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:50.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:50.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:50.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:50.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:50.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:50.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:50.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:51.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:51.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:51.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:51.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:51.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:51.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:51.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:51.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:51.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:51.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:51.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:51.323+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:51.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:51.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:51.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:51.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:51.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:52.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:52.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:52.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:52.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:51.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:52.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:52.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:52.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:52.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:52.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:52.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:52.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:52.353+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:52.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:52.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:52.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:52.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:53.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:53.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:53.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:53.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:53.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:53.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:53.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:53.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:53.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:53.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:53.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:53.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:53.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:53.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:53.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:53.397+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:53.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:53.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:53.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:53.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:54.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:54.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:54.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:54.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:54.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:54.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:54.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:54.036+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:54.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:54.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:54.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:54.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:54.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:54.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:54.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:54.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:54.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:54.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:54.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:54.704+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:55.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:55.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:55.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:55.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:55.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:55.018+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:55.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:55.019+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:55.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:58:55.456+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51947 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:58:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:55.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:55.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:55.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:55.448+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:56.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:55.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:56.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:55.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:56.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:55.741+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:56.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:55.742+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:56.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:56.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:56.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:56.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:56.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:56.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:56.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:56.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:56.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:56.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:56.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:56.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:56.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:56.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:56.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:56.701+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:57.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:57.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:57.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:57.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:56.970+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:57.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:57.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:57.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:57.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:57.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:57.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:57.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:57.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:58.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:58.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:58.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:58.456 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.012+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:58.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:58.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:58.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:58.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:58.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:58.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:58.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:58.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:58.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:58.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:58.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:58.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:59.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:59.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:59.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:58:59.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:58.972+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:58:59.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:59.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:59.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:59.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:59.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:59.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:58:59.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:58:59.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:59.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:59.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:59.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:58:59.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:58:59.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:00.312 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:59.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:00.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:59.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:00.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:58:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:59.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:00.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:58:59.999+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:00.585 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:00.464+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 51993 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:00.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:00.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:00.586 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:00.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:00.586 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:00.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:00.586 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:00.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:00.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:00.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:00.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:00.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:00.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:00.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:01.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:00.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:00.997+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:00.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:01.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:01.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:01.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:01.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:01.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:01.699+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:01.699+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:01.699+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:01.700+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:01.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:01.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:01.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:01.972+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:02.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:02.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:02.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:02.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:02.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:02.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:02.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:02.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:02.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:02.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:02.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:02.933+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:02.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:02.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:02.713+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:02.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:02.714+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:03.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:03.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:03.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:03.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:03.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:03.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:03.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:03.392+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:03.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:03.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:03.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:03.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:04.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:03.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:04.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:03.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:04.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:03.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:04.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:03.973+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:04.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:04.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:04.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:04.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:04.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:04.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:04.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:04.416+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:04.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:04.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:04.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:04.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:04.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:04.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:04.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:04.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:04.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:05.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:05.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:05.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:05.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:05.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:05.723 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:05.470+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52038 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:05.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:05.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:05.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:05.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:05.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:05.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:05.962+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:05.964+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:06.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:06.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:06.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:06.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:06.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:06.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:06.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:06.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:06.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:06.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:06.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:06.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:06.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:06.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:06.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:06.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:06.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:06.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:06.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:06.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:07.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:07.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:07.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:07.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:07.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:07.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:07.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:07.523+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:07.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:07.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:07.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:07.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:07.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:07.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:07.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:07.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:07.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:07.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:07.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:08.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:07.946+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:08.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:08.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:08.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:08.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:08.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:08.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:08.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:08.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:08.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:08.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:08.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:08.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:08.961+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:08.963+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:09.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:09.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:09.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:09.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:09.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:09.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:09.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:09.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:09.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:10.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:09.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:10.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:09.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:10.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:09.970+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:10.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:09.971+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:10.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:10.478+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52084 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:10.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:10.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:10.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:10.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:10.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:10.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:10.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:10.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:10.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:10.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:10.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:10.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:10.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:10.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:10.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:10.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:10.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:10.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:10.930+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:11.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:10.931+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:11.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:11.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:11.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:11.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:11.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:11.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:11.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:11.579+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:11.861 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:11.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:11.862 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:11.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:11.862 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:11.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:11.862 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:11.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:11.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:11.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:11.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:11.913+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:12.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:12.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:12.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:12.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:12.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:12.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:12.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:12.743+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:12.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:12.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:12.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:12.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:12.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:12.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:12.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:12.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:12.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:12.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:12.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:12.956+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:13.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:13.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:13.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:13.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:13.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:13.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:13.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:13.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:13.743+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:13.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:13.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:13.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:13.973+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:14.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:14.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:14.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:14.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:14.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:14.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:14.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:14.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:14.701+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:14.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:14.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:14.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:14.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:15.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:15.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:15.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:15.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:15.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:15.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:15.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:15.551+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:15.957 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:15.484+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52123 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:15.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:15.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:15.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:15.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:15.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:15.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:15.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:15.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:15.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:15.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:15.953+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:15.954+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:16.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:16.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:16.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:16.572+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:16.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:16.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:16.745+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:16.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:16.746+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:16.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:16.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:16.973+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:16.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:17.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:17.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:17.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:17.556+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:17.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:17.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:17.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:17.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:17.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:17.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:17.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:17.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:17.978+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:18.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:18.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:18.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:18.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:18.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:18.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:18.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:18.759+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:18.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:18.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:18.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:18.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:19.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:19.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:19.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:19.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:19.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:19.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:19.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:19.711+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:19.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:19.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:19.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:19.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:19.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:19.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:19.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:19.513+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:19.944+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:19.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:19.945+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:19.945+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:20.963 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:20.491+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52163 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:20.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:20.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:20.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:20.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:20.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:20.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:20.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:20.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:20.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:20.738+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:20.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:20.738+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:20.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:20.738+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:20.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:20.739+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:20.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:20.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:21.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:20.960+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:21.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:20.962+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:21.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:21.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:21.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:21.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:21.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:21.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:21.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:21.701+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:21.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:21.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:21.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:21.977+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:22.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:22.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:22.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:22.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:22.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:22.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:22.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:22.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:23.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:23.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:23.005+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:23.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:23.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:23.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:23.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:23.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:23.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:23.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:23.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:23.475+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:24.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:23.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:24.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:23.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:24.012 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:23.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:24.012 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:23.731+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:24.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:24.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:24.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:24.009+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:24.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:24.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:24.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:24.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:24.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:24.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:24.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:24.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:25.016 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:24.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:25.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:24.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:25.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:24.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:25.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:24.758+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.015+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:25.784 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:25.497+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52201 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:25.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:25.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:25.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:25.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:25.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:25.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:25.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:25.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:25.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:25.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:25.781+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:25.782+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:25.990+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:26.763 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:26.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:26.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:26.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:26.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:26.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:26.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:26.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:27.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:27.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:27.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:27.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:27.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:27.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:26.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:26.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:26.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:27.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:26.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:27.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:27.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:27.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:27.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:27.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:27.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:27.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:27.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:27.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:27.717+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:28.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:28.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:28.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:28.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:28.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:28.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:28.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:28.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:28.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:28.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:28.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:28.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:29.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:29.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:29.004+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:29.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:29.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:29.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:29.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:29.729+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:29.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:29.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:29.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:29.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:29.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:29.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:29.496+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:30.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:30.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:30.315 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:30.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:30.315 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:30.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:30.315 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:30.007+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:30.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:30.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:30.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:30.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:30.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:30.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:30.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:30.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:30.629 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:30.505+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52242 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:30.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:30.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:30.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:30.707+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:31.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:31.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:31.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:31.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:31.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:31.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:31.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:31.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:31.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:31.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:31.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:31.563+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:31.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:31.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:31.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:31.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:31.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:31.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:31.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:31.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:32.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:32.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:32.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:32.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:32.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:32.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:32.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:32.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:32.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:32.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:32.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:32.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:32.778+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:33.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:33.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:33.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:33.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:33.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:33.047+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:33.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:33.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:33.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:33.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:33.592+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:33.594+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:34.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:33.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:34.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:33.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:34.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:33.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:34.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:33.828+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:34.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:34.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:34.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:34.082+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:34.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:34.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:34.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:34.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:34.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:34.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:34.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:34.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:34.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:34.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:34.844+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:35.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:35.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:35.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:35.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:35.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:35.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:35.511+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52280 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:35.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:35.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:35.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:35.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:35.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:35.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:35.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:35.632+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:36.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:35.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:36.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:35.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:36.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:35.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:36.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:35.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:36.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:36.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:36.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:36.102+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:36.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:36.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:36.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:36.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:36.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:36.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:36.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:36.677+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:37.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:36.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:37.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:36.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:37.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:36.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:37.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:36.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:37.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:37.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:37.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:37.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:37.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:37.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:37.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:37.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:37.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:37.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:37.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:37.664+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:38.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:37.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:38.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:37.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:38.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:37.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:38.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:37.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:38.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:38.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:38.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:38.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:38.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:38.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:38.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:38.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:38.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:38.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:38.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:38.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:38.677+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:39.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:39.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:39.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:39.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:39.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:39.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:39.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:39.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:39.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:39.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:39.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:39.927+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:40.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:40.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:40.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:40.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:40.963 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:40.519+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52320 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:40.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:40.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:40.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:40.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:40.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:40.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:40.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:40.696+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:41.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:41.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:41.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:41.084+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:40.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:40.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:40.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:40.961+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:42.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:41.737+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:42.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:41.737+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:42.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:41.737+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:42.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:41.738+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:42.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:41.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:42.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:41.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:42.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:41.939+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:42.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:41.940+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:42.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:42.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:42.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:42.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:42.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:42.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:42.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:42.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:42.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:42.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:42.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:42.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:42.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:42.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:42.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:42.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:43.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:43.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:43.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:43.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:44.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:43.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:44.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:43.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:44.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:43.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:44.064 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:43.732+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:44.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:43.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:44.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:43.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:44.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:43.873+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:44.064 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:43.874+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:44.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:44.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:44.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:44.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:44.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:44.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:44.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:44.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:44.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:44.708+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:44.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:44.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:44.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:44.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:45.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:45.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:45.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:45.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:45.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:45.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:45.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:45.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:45.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:45.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:45.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:45.711+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:45.902 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:45.524+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52359 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:45.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:45.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:45.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:45.898+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:46.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:46.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:46.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:46.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:46.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:47.077 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:46.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:46.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:46.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:46.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:46.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:46.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:46.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:47.078 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:46.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:47.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:47.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:47.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:47.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:47.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:47.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:47.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:47.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:47.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:47.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:47.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:47.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:47.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:47.928+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:47.928+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:47.928+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:47.929+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:48.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:48.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:48.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:48.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:49.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:48.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:48.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:48.909+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:48.910+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:48.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:48.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:48.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:49.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:48.770+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:49.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:49.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:49.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:49.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:49.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:49.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:50.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:49.782+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:50.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:49.782+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:50.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:49.782+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:50.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:49.783+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:50.080 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:49.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:50.080 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:49.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:50.080 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:49.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:50.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:49.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:50.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:50.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:50.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:50.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:50.809 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:50.533+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52399 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:50.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:50.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:50.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:50.807+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:50.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:50.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:50.952+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:50.953+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:51.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:51.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:51.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:51.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:51.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:51.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:51.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:51.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:51.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:51.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:51.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:51.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:51.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:51.945+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:51.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:52.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:52.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:52.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:52.837+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:52.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:52.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:52.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:52.965+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:53.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:53.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:52.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:54.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:53.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:54.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:53.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:54.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:53.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:54.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:53.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:54.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:53.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:54.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:53.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:54.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:53.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:54.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:53.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:54.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:54.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:54.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:54.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:55.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:55.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:55.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:55.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:55.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:55.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:55.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:55.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:54.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:55.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:54.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:55.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:54.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:55.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:54.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:54.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:54.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:54.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:54.853+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:55.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T08:59:55.540+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52437 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T08:59:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:56.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:56.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:56.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:56.040+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:55.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:55.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:55.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:55.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:55.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:55.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:55.930+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:55.931+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:56.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:56.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:56.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:56.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:56.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:56.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:56.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:56.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:57.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:57.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:57.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:57.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:57.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:58.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:58.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:58.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:58.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:57.822+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:57.822+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:57.822+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:57.823+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:57.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:57.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:57.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:57.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:59.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:59.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:59.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T08:59:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T08:59:59.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T08:59:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:58.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:58.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:58.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T08:59:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:58.871+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:58.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:59.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:58.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:59.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:58.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T08:59:59.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:58.941+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:00.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:00.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:00.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:00.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:59.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:59.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:59.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T08:59:59.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:59.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:59.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:59.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 08:59:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T08:59:59.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:00.569 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:00.547+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52477 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:00.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:00.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:00.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:00.952+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:00.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:00.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:00.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:00.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:01.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:01.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:01.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:01.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:01.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:01.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:01.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:01.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:01.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:01.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:02.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:01.987+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:01.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:01.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:01.947+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:01.948+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:02.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:02.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:02.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:02.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:02.096+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:02.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:02.097+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:03.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:03.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:03.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:03.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:03.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:03.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:03.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:03.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:03.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:03.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:02.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:02.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:02.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:02.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:04.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:03.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:04.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:03.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:04.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:03.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:04.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:03.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:04.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:04.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:04.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:04.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:04.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:04.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:04.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:04.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:05.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:05.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:05.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:05.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:05.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:05.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:05.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:05.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:05.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:05.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:05.012+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:05.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:05.013+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:04.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:04.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:04.956+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:05.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:04.957+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:05.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:05.553+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52515 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:06.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:06.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:06.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:06.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:06.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:06.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:06.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:06.089+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:06.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:06.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:06.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:06.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:06.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:06.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:06.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:06.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:06.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:06.002+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:07.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:07.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:07.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:07.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:07.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:07.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:07.074+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:07.075+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:07.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:07.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:07.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:07.005+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:08.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:08.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:08.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:08.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:08.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:08.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:08.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:08.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:08.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:08.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:08.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:08.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:08.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:08.047+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:09.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:09.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:09.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:09.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:09.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:09.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:09.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:09.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:09.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:09.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:09.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:09.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:09.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:09.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:10.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:10.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:10.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:10.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:10.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:10.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:10.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:10.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:10.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:10.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:10.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:10.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:10.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:10.121+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:10.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:10.560+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52555 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:11.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:11.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:11.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:11.156+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:11.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:11.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:11.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:11.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:11.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:11.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:11.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:11.145+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:12.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:12.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:12.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:12.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:12.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:12.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:12.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:12.204+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:12.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:12.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:12.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:12.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:12.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:13.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:13.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:13.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:13.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:13.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:13.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:13.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:13.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:13.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:13.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:13.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:13.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:13.209+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:13.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:13.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:13.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:13.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:13.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:14.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:14.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:14.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:14.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:14.100+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:14.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:14.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:14.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:14.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:14.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:14.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:14.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:14.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:14.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:15.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:15.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:15.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:15.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:15.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:15.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:15.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:15.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:15.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:15.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:15.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:15.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:15.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:15.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:15.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:15.567+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52593 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:16.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:16.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:16.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:16.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:16.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:16.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:16.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:16.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:16.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:16.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:16.296+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:16.297+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:16.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:16.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:16.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:16.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:16.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:16.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:17.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:17.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:17.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:17.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:17.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:17.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:17.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:17.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:17.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:17.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:17.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:17.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:17.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:17.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:18.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:18.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:18.101+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:18.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:18.102+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:18.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:18.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:18.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:18.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:18.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:18.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:18.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:18.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:18.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:19.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:19.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:19.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:19.096+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:19.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:19.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:19.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:19.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:19.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:19.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:19.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:19.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:19.175+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:20.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:20.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:20.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:20.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:20.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:20.182+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:20.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:20.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:20.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:20.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:20.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:20.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:20.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:20.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:20.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:20.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:20.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:20.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:20.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:20.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:20.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:20.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:20.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:20.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:20.696 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:20.573+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52633 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:21.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:21.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:21.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:21.094+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:21.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:21.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:21.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:21.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:21.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:21.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:21.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:21.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:21.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:22.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:22.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:22.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:22.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:22.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:22.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:22.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:22.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:22.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:22.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:22.308+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:22.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:22.309+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:22.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:22.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:22.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:22.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:22.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:22.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:22.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:22.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:23.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:23.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:23.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:23.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:23.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:23.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:23.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:23.044+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:23.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:23.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:23.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:23.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:23.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:23.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:23.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:23.235+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:23.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:23.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:23.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:23.336+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:24.317 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:24.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:24.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:24.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:24.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:24.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:24.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:24.064+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:24.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:24.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:24.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:24.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:24.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:24.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:24.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:24.231+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:24.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:24.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:24.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:24.316+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:25.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:25.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:25.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:25.111+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:25.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:25.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:25.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:25.318+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:25.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:25.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:25.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:25.230+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:25.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:25.580+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52671 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:26.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:26.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:26.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:26.071+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:26.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:26.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:26.294+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:26.295+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:26.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:26.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:26.199+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:26.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:26.199+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:27.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:27.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:27.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:27.288+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:27.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:27.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:27.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:27.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:27.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:27.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:27.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:27.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:27.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:28.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:28.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:28.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:28.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:28.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:28.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:28.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:28.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:28.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:28.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:28.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:28.216+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:29.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:29.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:29.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:29.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:29.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:29.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:29.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:29.022+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:29.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:29.023+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:29.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:29.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:29.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:29.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:29.173+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:30.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:30.318 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:30.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.014+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:30.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.014+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:30.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:30.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:30.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:30.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:30.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:30.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:30.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:30.187+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:30.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:30.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:30.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:30.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:30.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:30.167+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:30.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:30.168+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:30.968 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:30.586+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52711 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:31.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:31.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:31.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:31.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:31.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:30.966+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:31.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:31.162+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:31.162+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:31.162+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:31.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:31.163+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:32.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:31.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:32.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:31.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:32.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:31.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:32.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:31.973+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:32.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:32.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:32.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:32.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:32.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:32.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:32.216+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:32.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:32.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:32.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:32.996+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:33.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:32.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:33.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:33.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:33.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:33.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:33.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:33.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:33.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:33.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:33.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:33.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:33.221+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:34.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:34.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:34.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:34.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:34.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:34.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:34.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:34.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:34.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:34.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:34.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:34.186+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:34.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:34.187+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:35.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:35.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:35.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:35.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:35.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:35.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:35.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:35.007+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:35.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:35.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:35.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:35.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:35.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:35.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:35.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:35.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:35.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:35.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:35.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:35.593+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52749 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:36.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:36.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:36.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:36.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:36.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:36.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:36.108+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:36.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:36.109+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:36.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:36.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:36.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:36.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:36.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:36.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:36.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:36.179+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:37.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:37.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:37.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:37.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:37.066+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:37.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:37.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:37.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:37.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:37.181+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:37.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:37.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:37.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:37.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:37.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:38.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:38.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:38.062+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:38.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:38.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:38.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:38.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:38.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:38.096+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:38.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:38.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:38.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:38.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:38.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:38.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:38.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:39.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:39.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:39.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:39.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:39.072+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:39.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:39.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:39.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:39.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:39.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:39.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:39.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:39.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:39.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:39.136+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:40.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:40.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:40.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:40.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:40.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:40.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:40.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:40.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:40.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:40.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:40.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:40.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:40.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:40.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:40.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:40.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:40.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:40.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:40.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:40.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:40.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:40.140+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:40.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:40.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:40.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:40.599+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52788 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:41.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:41.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:41.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:41.154+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:41.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:41.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:41.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:41.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:41.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:41.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:41.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:41.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:41.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:41.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:41.019+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:42.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:42.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:42.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:42.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:42.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:42.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:42.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:42.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:42.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:42.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:42.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:42.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:42.145+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:42.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:42.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:42.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:42.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:42.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:43.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:43.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:43.110+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:43.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:43.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:43.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:43.111+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:43.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:43.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:43.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:43.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:43.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:43.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:43.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:43.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:43.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:43.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:44.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:44.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:44.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:44.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:44.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:44.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:44.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:44.051+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:44.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:44.052+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:44.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:44.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:44.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:44.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:44.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:44.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:44.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:44.209+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:45.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:45.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:45.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:45.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:45.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:45.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:45.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:45.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:45.024+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:45.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:45.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:45.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:45.192+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:45.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:45.607+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52827 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:46.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:46.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:46.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:46.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:46.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:46.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:46.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:46.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:46.072+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:46.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:46.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:46.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:46.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:46.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:47.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:47.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:47.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:47.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:47.103+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:47.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:47.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:47.061+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:47.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:47.062+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:47.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:47.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:47.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:47.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:47.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:47.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:47.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:47.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:48.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:48.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:48.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:48.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:48.106+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:48.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:48.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:48.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:48.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:48.080+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:48.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:48.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:48.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:48.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:48.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:48.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:48.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:48.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:49.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:49.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:49.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:49.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:49.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:49.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:49.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:49.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:49.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:49.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:49.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:49.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:49.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:49.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:49.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:49.120+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:49.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:49.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:50.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:50.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:50.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:50.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:50.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:50.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:50.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:50.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:50.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:50.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:50.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:50.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:50.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:50.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:50.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:50.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:50.614+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52866 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:51.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:51.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:51.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:51.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:51.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:51.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:51.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:51.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:51.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:51.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:51.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:51.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:51.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:52.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:52.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:52.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:52.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:52.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:52.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:52.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:52.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:52.128+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:52.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:52.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:52.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:52.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:52.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:53.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:53.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:53.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:53.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:53.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:53.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:53.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:53.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:53.106+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:53.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:53.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:53.174+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:53.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:53.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:54.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:54.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:54.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:54.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:54.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:54.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:54.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:54.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:54.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:54.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:54.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:54.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:54.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:54.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:54.202+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:55.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:55.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:55.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:55.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:55.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:55.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:55.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:55.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:55.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:55.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:55.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:55.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:55.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:55.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:55.207+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:55.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:00:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:00:55.620+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52905 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:00:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:56.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:56.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:56.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:56.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:56.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:56.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:56.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:56.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:56.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:56.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:56.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:56.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:57.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:57.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:57.185+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:57.186+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:57.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:57.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:57.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:57.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:57.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:57.114+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:57.114+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:57.114+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:57.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:57.115+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:58.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:58.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:58.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:58.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:58.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:58.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:58.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:58.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:58.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:58.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:58.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:58.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:58.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:58.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:59.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:59.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:59.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:00:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:00:59.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:59.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:59.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:59.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:00:59.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:00:59.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:00:59.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:59.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:59.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:59.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:00:59.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:00:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:00:59.064+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:00.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:00.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:00.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:00.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:00.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:00.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:00.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:00.235+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:00.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:00.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:00.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:00.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:00.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:00.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:00.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:00.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:00.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:00.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:00.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:00.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:00.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:00.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:00.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:00.151+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:00.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:00.627+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52944 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:01.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:01.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:01.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:01.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:01.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:01.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:01.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:01.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:01.053+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:01.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:01.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:01.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:01.201+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:02.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:02.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:02.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:02.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:02.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:02.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:02.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:02.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:02.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:02.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:02.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:02.227+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:02.155+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:02.155+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:02.155+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:02.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:02.156+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:03.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:03.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:03.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:03.242+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:03.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:03.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:03.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:03.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:03.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:03.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:03.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:03.152+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:03.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:03.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:03.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:03.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:03.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:03.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:03.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:03.095+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:04.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:04.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:04.125+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:04.126+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:04.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:04.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:04.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:04.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:04.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:04.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:04.100+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:04.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:04.101+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:05.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:05.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:05.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:05.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:05.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:05.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:05.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:05.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:05.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:05.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:05.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:05.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:05.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:05.125+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:05.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:05.634+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 52983 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:06.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:06.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:06.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:06.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:06.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:06.093+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:06.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:06.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:06.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:06.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:06.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:06.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:06.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:06.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:06.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:07.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:07.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:07.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:07.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:07.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:07.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:07.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:07.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:07.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:07.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:07.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:07.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:07.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:07.123+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:08.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:08.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:08.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:08.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:08.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:08.157+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:08.366 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:08.158+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:08.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:08.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:08.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:08.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:08.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:08.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:08.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:08.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:08.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:08.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:08.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:08.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:09.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:09.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:09.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:09.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:09.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:09.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:09.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:09.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:09.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:09.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:09.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:09.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:09.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:09.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:09.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:09.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:10.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:10.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:10.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:10.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:10.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:10.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:10.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:10.147+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:10.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:10.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:10.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:10.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:09.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:10.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:10.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:10.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:10.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:10.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:10.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:10.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:10.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:10.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:10.641+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53022 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:11.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:11.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:11.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:11.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:11.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:11.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:11.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:11.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:11.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:11.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:11.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:11.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:11.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:11.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:11.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:11.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:11.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:11.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:12.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:12.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:12.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:12.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:12.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:12.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:12.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:12.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:12.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:12.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:12.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:12.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:12.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:12.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:12.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:12.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:12.353+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:13.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:13.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:13.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:13.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:13.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:13.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:13.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:13.147+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:13.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:13.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:13.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:13.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:13.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:13.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:13.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:13.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:13.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:13.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:13.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:13.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:14.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:14.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:14.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:14.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:14.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:14.098+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:14.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:14.099+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:14.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:14.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:14.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:14.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:14.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:14.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:14.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:14.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:14.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:14.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:14.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:14.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:15.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:15.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:15.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:15.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:15.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:15.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:15.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:15.094+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:15.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:15.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:15.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:15.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:15.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:15.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:15.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:15.131+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:15.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:15.648+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53061 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:15.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:15.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:15.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:15.432+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:16.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:16.057+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:16.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:16.058+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:16.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:16.058+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:16.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:16.058+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:16.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:16.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:16.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:16.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:16.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:16.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:16.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:16.165+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:16.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:16.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:16.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:16.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:17.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:17.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:17.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:17.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:17.080+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:17.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:17.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:17.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:17.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:17.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:17.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:17.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:17.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:17.369+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:18.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:18.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:18.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:18.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:18.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:18.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:18.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:18.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:18.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:18.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:18.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:18.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:18.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:18.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:18.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:18.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:18.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:18.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:18.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:18.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:18.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:18.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:18.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:18.413+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:19.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:19.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:19.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:19.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:19.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:19.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:19.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:19.055+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:19.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:19.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:19.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:19.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:19.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:19.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:19.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:19.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:19.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:19.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:19.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:19.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:20.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:20.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:20.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:20.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:20.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:20.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:20.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:20.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:20.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:20.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:20.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:20.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:20.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:20.072+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:20.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:20.073+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:20.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:20.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:20.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:20.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:20.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:20.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:20.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:20.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:20.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:20.655+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53100 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:21.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:21.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:21.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:21.083+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:21.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:21.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:21.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:21.093+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:21.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:21.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:21.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:21.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:22.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:22.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:22.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:22.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:22.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:22.111+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:22.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:22.113+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:22.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:22.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:22.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:22.320+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:22.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:23.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:23.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:23.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:23.304+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:23.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:23.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:23.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:23.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:23.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:23.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:23.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:23.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:23.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:23.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:24.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:24.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:24.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:24.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:24.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:24.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:24.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:24.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:24.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:24.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:24.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:24.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:24.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:24.276+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:24.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:24.277+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:25.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:25.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:25.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:25.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:25.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:25.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:25.162+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:25.163+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:25.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:25.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:25.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:25.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:25.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:25.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:25.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:25.661+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53139 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:26.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:26.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:26.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:26.039+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:26.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:26.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:26.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:26.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:26.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:26.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:26.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:26.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:26.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:26.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:26.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:27.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:27.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:27.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:27.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:27.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:27.232+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:27.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:27.233+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:27.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:27.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:27.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:27.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:27.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:27.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:27.352 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:27.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:27.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:27.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:27.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:27.349+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:28.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:28.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:28.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:28.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:28.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:28.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:28.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:28.258+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:28.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:28.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:28.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:28.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:28.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:28.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:28.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:28.095+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:28.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:28.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:28.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:28.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:29.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:29.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:29.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:29.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:29.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:29.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:29.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:29.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:29.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:29.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:29.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:29.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:29.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:29.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:29.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:29.145+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:29.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:29.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:29.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:29.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:30.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:30.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:30.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:30.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:30.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:30.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:30.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:30.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:30.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:30.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:30.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:30.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:30.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:30.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:30.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:30.178+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:30.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:30.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:30.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:30.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:30.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:30.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:30.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:30.431+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:30.691 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:30.668+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53178 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:31.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:31.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:31.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:31.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:31.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:31.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:31.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:31.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:31.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:31.869 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:31.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:31.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:31.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:31.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:31.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:31.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:31.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:32.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:32.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:32.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:32.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:32.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:32.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:32.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:32.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:32.186+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:32.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:32.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:32.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:32.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:33.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:33.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:33.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:33.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:33.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:33.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:33.200+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:33.201+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:33.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:33.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:33.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:33.520+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:34.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:34.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:34.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:34.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:34.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:34.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:34.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:34.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:34.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:34.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:34.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:34.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:34.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:34.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:34.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:34.232+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:34.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:34.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:34.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:34.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:35.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:35.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:35.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:35.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:35.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:35.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:35.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:35.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:35.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:35.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:35.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:35.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:35.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:35.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:35.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:35.246+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:35.796 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:35.674+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53218 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:35.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:35.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:35.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:35.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:35.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:35.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:35.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:35.542+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:36.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:36.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:36.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:36.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:36.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:36.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:36.211+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:36.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:36.212+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:36.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:36.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:36.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:36.552+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:37.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:37.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:37.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:37.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:37.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:37.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:37.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:37.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:37.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:37.254+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:37.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:37.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:37.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:37.519+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:38.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:38.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:38.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:38.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:38.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:38.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:38.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:38.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:38.198+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:38.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:38.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:38.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:38.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:39.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:39.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:39.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:39.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:39.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:39.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:39.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:39.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:39.532 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:39.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:39.532 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:39.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:39.532 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:39.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:39.533 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:39.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:39.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:39.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:39.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:39.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:40.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:40.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:40.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:40.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:40.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:40.238+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:40.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:40.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:40.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:40.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:40.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:40.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:40.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:40.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:40.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:40.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:40.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:40.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:40.681+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53257 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:41.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:41.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:41.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:41.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:41.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:41.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:41.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:41.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:41.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:41.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:41.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:41.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:41.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:41.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:42.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:42.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:42.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:42.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:42.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:42.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:42.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:42.290+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:42.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:42.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:42.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:42.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:43.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:43.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:43.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:43.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:43.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:43.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:43.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:43.274+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:43.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:43.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:43.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:43.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:43.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:44.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:44.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:44.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:44.277+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:44.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:44.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:44.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:44.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:44.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:44.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:44.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:44.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:45.535 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:45.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:45.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:45.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:45.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:45.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:45.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:45.493+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:45.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:45.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:45.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:45.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:45.264+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:45.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:45.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:45.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:45.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:45.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:45.809 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:45.687+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53296 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:46.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:46.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:46.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:46.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:46.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:46.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:46.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:46.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:46.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:46.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:46.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:46.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:47.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:47.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:47.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:47.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:47.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:47.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:47.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:47.287+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:47.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:47.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:47.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:47.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:47.361+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:48.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:48.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:48.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:48.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:48.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:48.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:48.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:48.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:48.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:48.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:48.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:48.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:48.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:48.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:48.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:49.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:49.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:49.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:49.311+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:49.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:49.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:49.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:49.373+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:49.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:49.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:49.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:49.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:49.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:49.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:49.462+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:50.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:50.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:50.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:50.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:50.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:50.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:50.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:50.416+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:50.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:50.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:50.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:50.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:50.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:50.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:50.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:50.314+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:50.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:50.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:50.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:50.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:50.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:50.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:50.625 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:50.378+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:50.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:50.694+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53335 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:51.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:51.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:51.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:51.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:51.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:51.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:51.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:51.273+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:51.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:51.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:51.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:51.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:51.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:51.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:51.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:51.405+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:51.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:51.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:51.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:51.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:51.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:51.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:51.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:51.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:52.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:52.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:52.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:52.311+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:52.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:52.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:52.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:52.413+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:52.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:52.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:52.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:52.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:52.413+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:53.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:53.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:53.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:53.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:53.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:53.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:53.309+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:53.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:53.310+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:53.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:53.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:53.372+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:53.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:53.373+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:54.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:54.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:54.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:54.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:54.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:54.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:54.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:54.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:54.342+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:54.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:54.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:54.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:54.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:54.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:55.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:55.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:55.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:55.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:55.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:55.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:55.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:55.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:55.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:55.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:55.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:55.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:55.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:55.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:55.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:55.348+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:55.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:55.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:55.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:55.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:55.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:55.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:55.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:55.450+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:55.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:01:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:01:55.701+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53374 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:01:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:56.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:56.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:56.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:56.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:56.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:56.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:56.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:56.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:56.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:56.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:56.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:56.498+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:56.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:56.499+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:57.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:57.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:57.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:57.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:57.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:57.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:57.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:57.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:57.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:57.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:57.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:57.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:57.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:57.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:57.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:57.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:57.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:57.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:57.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:58.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:58.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:58.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:58.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:58.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:58.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:58.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:58.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:58.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:58.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:58.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:58.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:58.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:58.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:58.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:58.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:58.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:58.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:58.520+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:58.521+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:59.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:59.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:59.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:01:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:01:59.335+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:59.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:59.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:59.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:01:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:01:59.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:01:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:59.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:59.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:59.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:59.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:59.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:01:59.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:01:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:01:59.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:00.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:00.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:00.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:00.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:00.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:00.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:00.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:00.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:00.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:00.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:00.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:00.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:00.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:00.238+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:00.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:00.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:00.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:00.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:00.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:00.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:00.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:00.441+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:00.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:00.707+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53413 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:01.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:01.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:01.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:01.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:01.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:01.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:01.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:01.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:01.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:01.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:01.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:01.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:01.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:01.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:01.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:01.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:01.314+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:02.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:02.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:02.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:02.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:02.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:02.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:02.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:02.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:02.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:02.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:02.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:02.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:02.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:02.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:02.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:02.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:02.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:02.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:03.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:03.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:03.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:03.507+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:03.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:03.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:03.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:03.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:03.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:03.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:03.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:03.297+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:03.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:03.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:03.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:03.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:03.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:03.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:03.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:03.303+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:04.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:04.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:04.330+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:04.330+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:04.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:04.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:04.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:04.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:04.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:04.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:04.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:04.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:04.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:04.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:04.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:04.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:04.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:05.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:05.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:05.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:05.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:05.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:05.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:05.717 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:05.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:05.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:05.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:05.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:05.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:05.354+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:05.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:05.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:05.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:05.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:05.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:05.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:05.714+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53452 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:06.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:06.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:06.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:06.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:06.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:06.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:06.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:06.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:06.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:06.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:06.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:06.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:06.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:06.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:06.516+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:07.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:07.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:07.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:07.360+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:07.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:07.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:07.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:07.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:07.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:07.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:07.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:07.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:07.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:07.516+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:08.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:08.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:08.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:08.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:08.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:08.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:08.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:08.320+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:08.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:08.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:08.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:08.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:08.497+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:09.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:09.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:09.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:09.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:09.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:09.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:09.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:09.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:09.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:09.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:09.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:09.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:10.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:10.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:10.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:10.278+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:10.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:10.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:10.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:10.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:10.296+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:10.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:10.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:10.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:10.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:10.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:10.501+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:10.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:10.501+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:11.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:10.721+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53491 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:11.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:11.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:11.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:11.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:11.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:11.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:11.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:11.231+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:11.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:11.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:11.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:11.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:11.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:11.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:11.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:11.279+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:11.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:11.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:11.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:11.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:11.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:11.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:11.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:11.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:12.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:12.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:12.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:12.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:12.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:12.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:12.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:12.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:12.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:12.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:12.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:12.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:12.484+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:13.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:13.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:13.217+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:13.218+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:13.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:13.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:13.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:13.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:13.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:13.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:13.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:13.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:13.504+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:14.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:14.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:14.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:14.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:14.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:14.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:14.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:14.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:14.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:14.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:14.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:14.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:14.533+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:15.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:15.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:15.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:15.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:15.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:15.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:15.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:15.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:15.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:15.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:15.728+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53530 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:15.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:15.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:15.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:15.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:15.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:15.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:15.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:15.525+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:16.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:16.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:16.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:16.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:16.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:16.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:16.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:16.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:16.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:16.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:16.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:16.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:16.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:16.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:16.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:16.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:16.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:16.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:16.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:16.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:17.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:17.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:17.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:17.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:17.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:17.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:17.228+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:17.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:17.228+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:17.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:17.228+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:17.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:17.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:17.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:17.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:17.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:17.498+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:18.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:18.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:18.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:18.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:18.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:18.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:18.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:18.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:18.225+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:18.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:18.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:18.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:18.508+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:19.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:19.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:19.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:19.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:19.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:19.259+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:19.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:19.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:19.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:19.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:19.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:19.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:19.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:19.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:19.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:19.231+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:19.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:19.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:19.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:19.536+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:20.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:20.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:20.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:20.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:20.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:20.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:20.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:20.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:20.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:20.737 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:20.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:20.738 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:20.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:20.738 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:20.535+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:20.738 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:20.536+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:21.187 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:20.735+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53569 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:21.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:21.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:21.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:21.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:21.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:21.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:21.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:21.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:21.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:21.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:21.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:21.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:21.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:21.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:21.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:21.501+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:22.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:22.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:22.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:22.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:22.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:22.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:22.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:22.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:22.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:22.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:22.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:22.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:22.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:23.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:23.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:23.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:23.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:23.220+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:23.388+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:23.389142+0000) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:23.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:23.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:23.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:23.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:23.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:23.311+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:23.754 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:23.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:23.755 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:23.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:23.755 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:23.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:23.755 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:23.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:24.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:23.888+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:23.889669+0000) 2024-02-20T09:02:24.332 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:24.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:24.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:24.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:24.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:24.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:24.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:24.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:24.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:24.079+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.080432+0000) 2024-02-20T09:02:24.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:24.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:24.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:24.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:24.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:24.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:24.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:24.288+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:24.389+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.390735+0000) 2024-02-20T09:02:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:24.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:24.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:24.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:24.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:24.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:24.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:25.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:24.888+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.889867+0000) 2024-02-20T09:02:25.332 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:25.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:25.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:25.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:25.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:25.078+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.080618+0000) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:25.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:25.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:25.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:25.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:25.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:25.272+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:25.389+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.390943+0000) 2024-02-20T09:02:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:25.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:25.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:25.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:25.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:26.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:25.738+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.740405+0000) 2024-02-20T09:02:26.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:25.738+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.740616+0000) 2024-02-20T09:02:26.083 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:25.740+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53604 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:26.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:25.738+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.740377+0000) 2024-02-20T09:02:26.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:25.888+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.890020+0000) 2024-02-20T09:02:26.332 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:26.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:26.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:26.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:26.144+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:26.078+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:26.080793+0000) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:26.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:26.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:26.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:26.234+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:26.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:26.235+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:26.389+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:26.391104+0000) 2024-02-20T09:02:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:26.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:26.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:26.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:26.509+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:27.085 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:26.890+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:26.892610+0000) 2024-02-20T09:02:27.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:27.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:27.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:27.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:27.104+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:27.082+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:27.083951+0000) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:27.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:27.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:27.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:27.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:27.396 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:27.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:27.391+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:27.393130+0000) 2024-02-20T09:02:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:27.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:27.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:27.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:27.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:28.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:27.893+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:27.895123+0000) 2024-02-20T09:02:28.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:28.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:28.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:28.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:28.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:28.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:28.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:28.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:28.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:28.085+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:28.086699+0000) 2024-02-20T09:02:28.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:28.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:28.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:28.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:28.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:28.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:28.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:28.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:28.394+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:28.395542+0000) 2024-02-20T09:02:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:28.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:28.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:28.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:28.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:28.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:29.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:28.895+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:28.896604+0000) 2024-02-20T09:02:29.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:29.087+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.088800+0000) 2024-02-20T09:02:29.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:29.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:29.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:29.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:29.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:29.072+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:29.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:29.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:29.396+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.397790+0000) 2024-02-20T09:02:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:29.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:29.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:29.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:29.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:29.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:29.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:29.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:29.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:30.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:29.895+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.896781+0000) 2024-02-20T09:02:30.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:30.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:30.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:30.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:30.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:30.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:30.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:30.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:30.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:30.088+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.088963+0000) 2024-02-20T09:02:30.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:30.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:30.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:30.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:30.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:30.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:30.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:30.318+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:30.397+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.398046+0000) 2024-02-20T09:02:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:30.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:30.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:30.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:30.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:31.082 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:30.748+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53644 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:31.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:30.746+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.747726+0000) 2024-02-20T09:02:31.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:30.746+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.747722+0000) 2024-02-20T09:02:31.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:30.746+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.747565+0000) 2024-02-20T09:02:31.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:30.896+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.896954+0000) 2024-02-20T09:02:31.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:31.088+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:31.089140+0000) 2024-02-20T09:02:31.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:31.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:31.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:31.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:31.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:31.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:31.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:31.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:31.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:31.397+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:31.398234+0000) 2024-02-20T09:02:31.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:31.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:31.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:31.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:31.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:31.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:31.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:31.351+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:31.616 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:31.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:31.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:31.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:31.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:31.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:31.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:31.473+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:32.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:32.090+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:32.091675+0000) 2024-02-20T09:02:32.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:31.898+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:31.899450+0000) 2024-02-20T09:02:32.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:32.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:32.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:32.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:32.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:32.083+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:32.201 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:32.084+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:32.399+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:32.400846+0000) 2024-02-20T09:02:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:32.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:32.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:32.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:32.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:32.396+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:32.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:32.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:32.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:32.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:32.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:32.492+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:32.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:32.492+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:33.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:32.901+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:32.902299+0000) 2024-02-20T09:02:33.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:33.094+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:33.095025+0000) 2024-02-20T09:02:33.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:33.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:33.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:33.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:33.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:33.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:33.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:33.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:33.403+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:33.404276+0000) 2024-02-20T09:02:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:33.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:33.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:33.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:33.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:33.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:33.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:33.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:33.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:34.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:33.903+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:33.904372+0000) 2024-02-20T09:02:34.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:34.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:34.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:34.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:34.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:34.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:34.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:34.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:34.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:34.095+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.096847+0000) 2024-02-20T09:02:34.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:34.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:34.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:34.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:34.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:34.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:34.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:34.405+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.406344+0000) 2024-02-20T09:02:34.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:34.375+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:34.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:34.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:34.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:34.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:34.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:34.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:34.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:34.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:34.903+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.904555+0000) 2024-02-20T09:02:35.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:35.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:35.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:35.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:35.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:35.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:35.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:35.064+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:35.334 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:35.096+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.097034+0000) 2024-02-20T09:02:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:35.405+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.406525+0000) 2024-02-20T09:02:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:35.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:35.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:35.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:35.340+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:35.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:35.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:35.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:35.475+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:36.098 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:35.756+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53683 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:36.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:35.753+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.754522+0000) 2024-02-20T09:02:36.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:35.752+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.753512+0000) 2024-02-20T09:02:36.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:35.753+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.754817+0000) 2024-02-20T09:02:36.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:35.903+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.904760+0000) 2024-02-20T09:02:36.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:36.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:36.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:36.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:36.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:36.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:36.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:36.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:36.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:36.096+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:36.097200+0000) 2024-02-20T09:02:36.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:36.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:36.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:36.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:36.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:36.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:36.405+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:36.406716+0000) 2024-02-20T09:02:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:36.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:36.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:36.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:36.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:36.507+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:37.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:36.907+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:36.908364+0000) 2024-02-20T09:02:37.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:37.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:37.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:37.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:37.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:37.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:37.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:37.138+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:37.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:37.099+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:37.100235+0000) 2024-02-20T09:02:37.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:37.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:37.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:37.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:37.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:37.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:37.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:37.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:37.407+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:37.408840+0000) 2024-02-20T09:02:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:37.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:37.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:37.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:37.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:38.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:37.910+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:37.911898+0000) 2024-02-20T09:02:38.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:38.102+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:38.103554+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:38.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:38.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:38.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:38.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:38.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:38.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:38.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:38.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:38.410+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:38.411477+0000) 2024-02-20T09:02:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:38.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:38.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:38.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:38.534+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:38.535+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:39.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:38.912+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:38.913611+0000) 2024-02-20T09:02:39.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:39.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:39.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:39.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:39.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:39.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:39.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:39.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:39.363 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:39.104+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.105481+0000) 2024-02-20T09:02:39.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:39.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:39.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:39.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:39.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:39.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:39.412+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.412950+0000) 2024-02-20T09:02:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:39.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:39.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:39.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:39.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:39.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:39.568+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:40.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:39.912+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.913782+0000) 2024-02-20T09:02:40.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:40.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:40.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:40.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:40.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:40.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:40.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:40.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:40.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:40.104+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.105650+0000) 2024-02-20T09:02:40.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:40.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:40.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:40.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:40.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:40.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:40.412+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.413126+0000) 2024-02-20T09:02:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:40.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:40.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:40.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:40.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:40.574+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:41.106 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:40.762+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53722 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:41.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:40.760+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.761884+0000) 2024-02-20T09:02:41.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:40.760+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.761832+0000) 2024-02-20T09:02:41.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:40.760+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.761811+0000) 2024-02-20T09:02:41.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:40.913+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.913953+0000) 2024-02-20T09:02:41.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:41.104+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:41.105835+0000) 2024-02-20T09:02:41.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:41.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:41.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:41.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:41.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:41.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:41.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:41.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:41.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:41.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:41.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:41.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:41.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:41.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:41.412+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:41.413328+0000) 2024-02-20T09:02:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:41.385+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:41.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:41.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:41.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:41.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:42.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:41.916+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:41.917227+0000) 2024-02-20T09:02:42.417 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:42.107+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:42.108504+0000) 2024-02-20T09:02:42.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:42.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:42.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:42.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:42.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:42.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:42.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:42.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:42.415+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:42.416117+0000) 2024-02-20T09:02:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:42.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:42.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:42.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:42.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:42.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:42.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:42.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:42.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:43.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:42.918+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:42.919461+0000) 2024-02-20T09:02:43.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:43.110+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:43.111673+0000) 2024-02-20T09:02:43.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:43.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:43.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:43.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:43.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:43.183+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:43.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:43.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:43.419+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:43.420353+0000) 2024-02-20T09:02:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:43.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:43.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:43.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:43.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:43.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:43.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:43.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:43.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:44.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:43.920+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:43.921159+0000) 2024-02-20T09:02:44.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:44.112+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.113699+0000) 2024-02-20T09:02:44.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:44.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:44.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:44.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:44.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:44.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:44.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:44.154+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:44.421+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.422449+0000) 2024-02-20T09:02:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:44.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:44.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:44.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:44.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:44.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:44.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:44.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:44.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:45.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:44.920+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.921335+0000) 2024-02-20T09:02:45.424 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:45.112+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.113893+0000) 2024-02-20T09:02:45.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:45.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:45.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:45.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:45.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:45.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:45.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:45.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:45.421+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.422651+0000) 2024-02-20T09:02:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:45.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:45.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:45.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:14.027925+0000 (oldest deadline 2024-02-20T08:52:34.027925+0000) 2024-02-20T09:02:45.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:45.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:45.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:45.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:45.567+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:45.568+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:46.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:45.767+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.768551+0000) 2024-02-20T09:02:46.110 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:45.767+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.768609+0000) 2024-02-20T09:02:46.110 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:45.770+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53761 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:46.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:45.767+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.768297+0000) 2024-02-20T09:02:46.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:45.920+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.921515+0000) 2024-02-20T09:02:46.424 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:46.113+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:46.114059+0000) 2024-02-20T09:02:46.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:46.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:46.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:46.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:46.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:46.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:46.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:46.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:46.421+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:46.422831+0000) 2024-02-20T09:02:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:46.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:46.595+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:46.595+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:46.595+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:46.596+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:47.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:46.922+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:46.923138+0000) 2024-02-20T09:02:47.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:47.116+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:47.116956+0000) 2024-02-20T09:02:47.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:47.392+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:47.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:47.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:47.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:47.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:47.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:47.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:47.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:47.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:47.424+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:47.425354+0000) 2024-02-20T09:02:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:47.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:47.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:47.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:47.585+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:48.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:47.924+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:47.925575+0000) 2024-02-20T09:02:48.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:48.118+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:48.119355+0000) 2024-02-20T09:02:48.377 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:48.374+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:48.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:48.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:48.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:48.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:48.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:48.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:10.479199+0000 (oldest deadline 2024-02-20T08:52:30.479199+0000) 2024-02-20T09:02:48.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:48.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:48.426+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:48.427725+0000) 2024-02-20T09:02:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:48.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:48.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:48.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:48.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:49.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:48.926+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:48.927304+0000) 2024-02-20T09:02:49.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:49.218+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:49.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:49.120+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.121199+0000) 2024-02-20T09:02:49.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:49.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:49.428+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.429830+0000) 2024-02-20T09:02:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:49.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:49.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:49.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:49.578+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:50.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:49.926+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.927491+0000) 2024-02-20T09:02:50.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:50.202+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:50.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:50.120+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.121366+0000) 2024-02-20T09:02:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:50.429+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.429975+0000) 2024-02-20T09:02:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:50.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:50.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:50.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:50.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:50.624+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:51.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:50.774+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.775895+0000) 2024-02-20T09:02:51.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:50.773+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.774873+0000) 2024-02-20T09:02:51.114 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:50.776+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53800 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:51.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:50.773+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.774762+0000) 2024-02-20T09:02:51.114 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:50.926+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.927641+0000) 2024-02-20T09:02:51.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:51.222+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:51.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:51.120+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:51.121530+0000) 2024-02-20T09:02:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:51.429+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:51.430157+0000) 2024-02-20T09:02:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:51.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:51.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:51.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:51.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:51.641+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:52.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:51.929+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:51.930220+0000) 2024-02-20T09:02:52.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:52.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:52.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:52.124+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:52.124967+0000) 2024-02-20T09:02:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:52.431+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:52.432485+0000) 2024-02-20T09:02:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:52.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:52.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:52.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:52.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:52.654+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:53.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:52.931+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:52.932646+0000) 2024-02-20T09:02:53.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:53.242+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:53.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:53.127+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:53.128348+0000) 2024-02-20T09:02:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:53.434+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:53.435566+0000) 2024-02-20T09:02:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:53.495+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:53.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:53.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:53.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:52:18.780142+0000 (oldest deadline 2024-02-20T08:52:38.780142+0000) 2024-02-20T09:02:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:53.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:54.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:53.934+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:53.934956+0000) 2024-02-20T09:02:54.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:54.256+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:54.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:54.129+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.130703+0000) 2024-02-20T09:02:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:54.436+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.437822+0000) 2024-02-20T09:02:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:54.453+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:54.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:55.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:54.934+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.935119+0000) 2024-02-20T09:02:55.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:55.270+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:55.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:55.129+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.130874+0000) 2024-02-20T09:02:55.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:55.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:55.437+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.437974+0000) 2024-02-20T09:02:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:55.631+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:56.117 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:02:55.782+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53839 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:02:56.117 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:55.781+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.781946+0000) 2024-02-20T09:02:56.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:55.780+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.781538+0000) 2024-02-20T09:02:56.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:55.781+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.782667+0000) 2024-02-20T09:02:56.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:55.934+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.935286+0000) 2024-02-20T09:02:56.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:56.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:56.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:56.130+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:56.131061+0000) 2024-02-20T09:02:56.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:56.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:56.437+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:56.438149+0000) 2024-02-20T09:02:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:56.612+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:57.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:56.936+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:56.937921+0000) 2024-02-20T09:02:57.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:57.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:57.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:57.133+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:57.134630+0000) 2024-02-20T09:02:57.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:57.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:57.439+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:57.440550+0000) 2024-02-20T09:02:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:57.566+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:58.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:57.940+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:57.940956+0000) 2024-02-20T09:02:58.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:58.310+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:58.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:58.136+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:58.137403+0000) 2024-02-20T09:02:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:58.442+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:58.443111+0000) 2024-02-20T09:02:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:58.466+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:58.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:59.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:58.941+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:58.942653+0000) 2024-02-20T09:02:59.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:59.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:02:59.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:59.138+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.139383+0000) 2024-02-20T09:02:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:02:59.443+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.444825+0000) 2024-02-20T09:02:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:02:59.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:59.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:00.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:02:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:02:59.942+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.943143+0000) 2024-02-20T09:03:00.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:00.138+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.139587+0000) 2024-02-20T09:03:00.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:00.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:00.444+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.444961+0000) 2024-02-20T09:03:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:00.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:00.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:01.120 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:00.789+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53878 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:01.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:00.787+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.788329+0000) 2024-02-20T09:03:01.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:00.788+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.789295+0000) 2024-02-20T09:03:01.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:00.788+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.789572+0000) 2024-02-20T09:03:01.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:00.942+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.943323+0000) 2024-02-20T09:03:01.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:01.346+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:01.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:01.138+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:01.139767+0000) 2024-02-20T09:03:01.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:01.444+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:01.445137+0000) 2024-02-20T09:03:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:01.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:01.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:02.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:01.944+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:01.945396+0000) 2024-02-20T09:03:02.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:02.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:02.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:02.141+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:02.141968+0000) 2024-02-20T09:03:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:02.446+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:02.447300+0000) 2024-02-20T09:03:02.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:02.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:02.608+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:03.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:02.946+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:02.947478+0000) 2024-02-20T09:03:03.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:03.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:03.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:03.144+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:03.145317+0000) 2024-02-20T09:03:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:03.448+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:03.449643+0000) 2024-02-20T09:03:03.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:03.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:03.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:03.604+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:04.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:03.948+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:03.949804+0000) 2024-02-20T09:03:04.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:04.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:04.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:04.146+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.147534+0000) 2024-02-20T09:03:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:04.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:04.450+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.451211+0000) 2024-02-20T09:03:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:04.616+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:05.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:04.949+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.949953+0000) 2024-02-20T09:03:05.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:05.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:05.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:05.146+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.147718+0000) 2024-02-20T09:03:05.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:05.450+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.451394+0000) 2024-02-20T09:03:05.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:05.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:05.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:05.793+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.794914+0000) 2024-02-20T09:03:05.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:05.794+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.795596+0000) 2024-02-20T09:03:05.920 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:05.796+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53917 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:05.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:05.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:05.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:05.793+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.794580+0000) 2024-02-20T09:03:06.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:05.949+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.950128+0000) 2024-02-20T09:03:06.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:06.146+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:06.147910+0000) 2024-02-20T09:03:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:06.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:06.450+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:06.451590+0000) 2024-02-20T09:03:06.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:06.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:06.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:06.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:07.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:06.951+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:06.952633+0000) 2024-02-20T09:03:07.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:07.268+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:07.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:07.149+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:07.150294+0000) 2024-02-20T09:03:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:07.453+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:07.454358+0000) 2024-02-20T09:03:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:07.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:07.612+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:08.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:07.954+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:07.955172+0000) 2024-02-20T09:03:08.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:08.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:08.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:08.151+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:08.152811+0000) 2024-02-20T09:03:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:08.455+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:08.456893+0000) 2024-02-20T09:03:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:08.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:08.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:09.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:08.955+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:08.956837+0000) 2024-02-20T09:03:09.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:09.153+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.154275+0000) 2024-02-20T09:03:09.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:09.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:09.458+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.459169+0000) 2024-02-20T09:03:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:09.498+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:09.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:09.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:10.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:09.956+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.957038+0000) 2024-02-20T09:03:10.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:10.153+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.154458+0000) 2024-02-20T09:03:10.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:10.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:10.458+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.459347+0000) 2024-02-20T09:03:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:10.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:10.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:10.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:10.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:10.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:10.535+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:11.128 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:10.802+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.803121+0000) 2024-02-20T09:03:11.128 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:10.804+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53956 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:11.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:10.800+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.801714+0000) 2024-02-20T09:03:11.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:10.801+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.802137+0000) 2024-02-20T09:03:11.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:10.956+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.957202+0000) 2024-02-20T09:03:11.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:11.234+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:11.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:11.153+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:11.154635+0000) 2024-02-20T09:03:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:11.458+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:11.459506+0000) 2024-02-20T09:03:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:11.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:11.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:11.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:11.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:11.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:11.534+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:12.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:11.959+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:11.959961+0000) 2024-02-20T09:03:12.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:12.156+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:12.157074+0000) 2024-02-20T09:03:12.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:12.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:12.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:12.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:12.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:12.284+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:12.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:12.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:12.461+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:12.462024+0000) 2024-02-20T09:03:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:12.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:12.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:12.526+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:12.526+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:12.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:13.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:12.961+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:12.962238+0000) 2024-02-20T09:03:13.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:13.158+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:13.159457+0000) 2024-02-20T09:03:13.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:13.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:13.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:13.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:13.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:13.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:13.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:13.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:13.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:13.463+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:13.464653+0000) 2024-02-20T09:03:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:13.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:13.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:13.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:13.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:13.554+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:14.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:13.962+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:13.963828+0000) 2024-02-20T09:03:14.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:14.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:14.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:14.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:14.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:14.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:14.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:14.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:14.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:14.160+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.161264+0000) 2024-02-20T09:03:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:14.465+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.466190+0000) 2024-02-20T09:03:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:14.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:14.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:14.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:14.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:14.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:14.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:14.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:14.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:14.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:15.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:14.963+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.963956+0000) 2024-02-20T09:03:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:15.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:15.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:15.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:15.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:15.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:15.160+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.161425+0000) 2024-02-20T09:03:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:15.465+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.466366+0000) 2024-02-20T09:03:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:15.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:15.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:15.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:15.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:15.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:15.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:15.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:15.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:15.631+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:16.131 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:15.808+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.809195+0000) 2024-02-20T09:03:16.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:15.809+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.810535+0000) 2024-02-20T09:03:16.132 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:15.812+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 53995 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:16.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:15.811+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.812190+0000) 2024-02-20T09:03:16.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:15.963+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.964419+0000) 2024-02-20T09:03:16.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:16.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:16.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:16.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:16.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:16.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:16.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:16.371+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:16.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:16.160+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:16.161573+0000) 2024-02-20T09:03:16.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:16.465+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:16.466547+0000) 2024-02-20T09:03:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:16.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:16.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:16.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:16.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:16.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:16.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:16.650+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:16.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:16.651+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:17.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:16.966+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:16.966970+0000) 2024-02-20T09:03:17.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:17.163+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:17.164347+0000) 2024-02-20T09:03:17.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:17.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:17.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:17.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:17.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:17.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:17.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:17.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:17.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:17.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:17.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:17.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:17.469+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:17.470026+0000) 2024-02-20T09:03:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:17.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:17.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:17.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:17.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:17.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:18.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:18.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:18.166932+0000) 2024-02-20T09:03:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:17.968+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:17.969496+0000) 2024-02-20T09:03:18.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:18.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:18.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:18.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:18.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:18.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:18.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:18.366+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:18.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:18.470+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:18.472221+0000) 2024-02-20T09:03:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:18.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:18.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:18.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:18.578+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:18.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:18.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:18.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:18.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:19.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:19.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.168428+0000) 2024-02-20T09:03:19.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:18.969+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:18.971887+0000) 2024-02-20T09:03:19.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:19.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:19.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:19.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:19.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:19.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:19.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:19.329+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:19.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:19.471+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.473812+0000) 2024-02-20T09:03:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:19.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:19.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:19.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:19.587+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:19.665+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:19.665+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:19.665+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:19.666+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:20.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:20.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.168622+0000) 2024-02-20T09:03:20.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:19.970+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.972077+0000) 2024-02-20T09:03:20.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:20.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:20.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:20.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:20.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:20.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:20.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:20.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:20.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:20.472+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.473973+0000) 2024-02-20T09:03:20.939 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:20.817+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54034 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:20.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:20.815+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.817800+0000) 2024-02-20T09:03:20.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:20.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:20.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:20.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:20.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:20.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:20.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:20.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:20.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:20.815+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.817914+0000) 2024-02-20T09:03:20.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:20.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:20.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:20.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:20.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:20.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:20.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:20.637+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:20.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:20.815+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.817844+0000) 2024-02-20T09:03:21.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:21.167+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:21.168858+0000) 2024-02-20T09:03:21.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:20.970+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.972288+0000) 2024-02-20T09:03:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:21.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:21.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:21.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:21.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:21.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:21.472+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:21.474169+0000) 2024-02-20T09:03:21.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:21.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:21.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:21.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:21.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:21.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:21.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:21.555+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:21.776 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:21.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:21.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:21.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:21.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:21.628+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:21.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:21.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:22.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:21.973+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:21.974203+0000) 2024-02-20T09:03:22.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:22.169+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:22.170867+0000) 2024-02-20T09:03:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:22.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:22.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:22.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:22.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:22.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:22.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:22.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:22.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:22.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:22.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:22.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:22.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:22.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:22.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:22.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:22.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:22.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:22.475+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:22.476675+0000) 2024-02-20T09:03:23.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:22.976+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:22.976924+0000) 2024-02-20T09:03:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:23.172+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:23.173500+0000) 2024-02-20T09:03:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:23.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:23.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:23.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:23.359+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:23.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:23.478+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:23.479157+0000) 2024-02-20T09:03:23.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:23.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:23.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:23.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:23.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:23.561+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:23.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:23.562+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:23.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:23.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:23.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:23.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:23.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:23.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:23.761 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:23.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:24.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:23.978+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:23.979402+0000) 2024-02-20T09:03:24.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:24.174+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.175012+0000) 2024-02-20T09:03:24.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:24.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:24.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:24.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:24.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:24.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:24.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:24.557+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:24.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:24.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:24.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:24.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:24.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:24.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:24.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:24.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:24.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:24.479+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.480855+0000) 2024-02-20T09:03:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:24.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:24.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:24.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:24.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:25.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:25.174+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.175216+0000) 2024-02-20T09:03:25.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:24.978+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.979582+0000) 2024-02-20T09:03:25.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:25.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:25.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:25.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:25.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:25.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:25.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:25.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:25.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:25.480+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.481035+0000) 2024-02-20T09:03:25.947 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:25.825+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54073 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:25.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:25.822+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.823634+0000) 2024-02-20T09:03:25.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:25.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:25.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:25.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:25.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:25.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:25.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:25.679+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:25.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:25.823+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.824383+0000) 2024-02-20T09:03:25.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:25.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:25.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:25.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:25.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:25.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:25.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:25.607+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:25.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:25.823+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.824558+0000) 2024-02-20T09:03:26.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:26.174+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:26.175389+0000) 2024-02-20T09:03:26.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:25.978+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.979760+0000) 2024-02-20T09:03:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:26.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:26.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:26.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:26.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:26.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:26.480+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:26.481210+0000) 2024-02-20T09:03:26.776 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:26.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:26.776 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:26.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:26.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:26.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:26.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:26.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:26.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:26.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:26.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:26.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:26.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:26.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:26.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:26.559+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:27.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:26.981+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:26.982200+0000) 2024-02-20T09:03:27.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:27.177+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:27.177943+0000) 2024-02-20T09:03:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:27.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:27.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:27.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:27.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:27.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:27.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:27.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:27.653+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:27.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:27.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:27.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:27.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:27.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:27.483+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:27.484525+0000) 2024-02-20T09:03:28.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:27.984+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:27.985473+0000) 2024-02-20T09:03:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:28.179+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:28.180464+0000) 2024-02-20T09:03:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:28.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:28.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:28.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:28.367+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:28.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:28.487+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:28.488087+0000) 2024-02-20T09:03:28.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:28.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:28.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:28.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:28.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:28.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:28.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:28.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:28.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:28.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:29.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:28.986+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:28.987092+0000) 2024-02-20T09:03:29.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:29.181+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.182076+0000) 2024-02-20T09:03:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:29.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:29.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:29.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:29.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:29.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:29.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:29.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:29.638+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:29.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:29.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:29.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:29.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:29.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:29.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:29.489+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.490493+0000) 2024-02-20T09:03:30.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:29.986+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.987266+0000) 2024-02-20T09:03:30.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:30.181+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.182227+0000) 2024-02-20T09:03:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:30.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:30.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:30.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:30.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:30.489+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.490645+0000) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:30.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:30.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:30.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:30.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:30.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:30.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:30.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:30.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:30.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:31.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:30.829+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.830701+0000) 2024-02-20T09:03:31.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:30.829+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.830643+0000) 2024-02-20T09:03:31.141 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:30.832+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54112 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:31.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:30.829+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.830757+0000) 2024-02-20T09:03:31.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:30.986+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.987454+0000) 2024-02-20T09:03:31.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:31.181+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:31.182393+0000) 2024-02-20T09:03:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:31.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:31.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:31.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:31.455+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:31.489+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:31.490847+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:31.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:31.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:31.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:31.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:31.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:31.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:31.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:31.475+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:32.143 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:31.989+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:31.990766+0000) 2024-02-20T09:03:32.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:32.183+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:32.184788+0000) 2024-02-20T09:03:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:32.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:32.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:32.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:32.557+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:32.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:32.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:32.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:32.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:32.492+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:32.492947+0000) 2024-02-20T09:03:32.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:32.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:32.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:32.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:32.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:32.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:32.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:32.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:33.055 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:32.992+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:32.993450+0000) 2024-02-20T09:03:33.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:33.186+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:33.187331+0000) 2024-02-20T09:03:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:33.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:33.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:33.456+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:33.457+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:33.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:33.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:33.573+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:33.574+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:33.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:33.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:33.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:33.461+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:33.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:33.494+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:33.495676+0000) 2024-02-20T09:03:34.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:33.994+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:33.994953+0000) 2024-02-20T09:03:34.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:34.188+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.189745+0000) 2024-02-20T09:03:34.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:34.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:34.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:34.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:34.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:34.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:34.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:34.475+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:34.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:34.496+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.497400+0000) 2024-02-20T09:03:34.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:34.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:34.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:34.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:34.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:34.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:34.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:34.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:34.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:34.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:34.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:34.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:34.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:34.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:34.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:34.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:35.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:34.994+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.995118+0000) 2024-02-20T09:03:35.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:35.189+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.190102+0000) 2024-02-20T09:03:35.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:35.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:35.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:35.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:35.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:35.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:35.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:35.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:35.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:35.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:35.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:35.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:35.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:35.470+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:35.496+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.497609+0000) 2024-02-20T09:03:36.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:35.836+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.837025+0000) 2024-02-20T09:03:36.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:35.836+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.837559+0000) 2024-02-20T09:03:36.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:35.836+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.837107+0000) 2024-02-20T09:03:36.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:35.994+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.995321+0000) 2024-02-20T09:03:36.129 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:35.840+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54151 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:36.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:36.189+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:36.190273+0000) 2024-02-20T09:03:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:36.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:36.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:36.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:36.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:36.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:36.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:36.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:36.467+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:36.496+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:36.497799+0000) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:36.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:36.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:36.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:36.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:36.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:37.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:36.996+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:36.997803+0000) 2024-02-20T09:03:37.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:37.191+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:37.192400+0000) 2024-02-20T09:03:37.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:37.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:37.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:37.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:37.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:37.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:37.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:37.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:37.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:37.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:37.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:37.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:37.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:37.475+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:37.500+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:37.501022+0000) 2024-02-20T09:03:38.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:37.999+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:38.000499+0000) 2024-02-20T09:03:38.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:38.194+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:38.195623+0000) 2024-02-20T09:03:38.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:38.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:38.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:38.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:38.437+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:38.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:38.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:38.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:38.585+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:38.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:38.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:38.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:38.499+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:38.502+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:38.503829+0000) 2024-02-20T09:03:39.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:39.001+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.002744+0000) 2024-02-20T09:03:39.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:39.196+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.197167+0000) 2024-02-20T09:03:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:39.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:39.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:39.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:39.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:39.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:39.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:39.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:39.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:39.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:39.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:39.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:39.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:39.469+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:39.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:39.505+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.506116+0000) 2024-02-20T09:03:40.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:40.001+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.002923+0000) 2024-02-20T09:03:40.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:40.196+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.197351+0000) 2024-02-20T09:03:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:40.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:40.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:40.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:40.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:40.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:40.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:40.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:40.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:40.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:40.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:40.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:40.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:40.505+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.506305+0000) 2024-02-20T09:03:41.147 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:40.846+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54190 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:41.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:40.845+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.846558+0000) 2024-02-20T09:03:41.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:40.844+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.845168+0000) 2024-02-20T09:03:41.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:40.845+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.846325+0000) 2024-02-20T09:03:41.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:41.002+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:41.003112+0000) 2024-02-20T09:03:41.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:41.196+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:41.197522+0000) 2024-02-20T09:03:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:41.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:41.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:41.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:41.486+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:41.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:41.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:41.508+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:41.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:41.509+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:41.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:41.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:41.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:41.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:41.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:41.447+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:41.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:41.505+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:41.506491+0000) 2024-02-20T09:03:42.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:42.005+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:42.006512+0000) 2024-02-20T09:03:42.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:42.199+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:42.200785+0000) 2024-02-20T09:03:42.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:42.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:42.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:42.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:42.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:42.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:42.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:42.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:42.552+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:42.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:42.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:42.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:42.487+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:42.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:42.507+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:42.508875+0000) 2024-02-20T09:03:43.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:43.008+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:43.009905+0000) 2024-02-20T09:03:43.424 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:43.203+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:43.204067+0000) 2024-02-20T09:03:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:43.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:43.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:43.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:43.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:43.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:43.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:43.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:43.574+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:43.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:43.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:43.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:43.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:43.510+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:43.511450+0000) 2024-02-20T09:03:44.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:44.204+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.205816+0000) 2024-02-20T09:03:44.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:44.010+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.011582+0000) 2024-02-20T09:03:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:44.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:44.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:44.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:44.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:44.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:44.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:44.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:44.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:44.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:44.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:44.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:44.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:44.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:44.513+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.513930+0000) 2024-02-20T09:03:45.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:45.010+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.011761+0000) 2024-02-20T09:03:45.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:45.205+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.206057+0000) 2024-02-20T09:03:45.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:45.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:45.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:45.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:45.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:45.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:45.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:45.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:45.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:45.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:45.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:45.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:45.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:45.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:45.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:45.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:45.513+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.514130+0000) 2024-02-20T09:03:46.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:45.852+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.853728+0000) 2024-02-20T09:03:46.151 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:45.853+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54229 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:46.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:45.851+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.852901+0000) 2024-02-20T09:03:46.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:45.850+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.851260+0000) 2024-02-20T09:03:46.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:46.011+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:46.011954+0000) 2024-02-20T09:03:46.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:46.205+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:46.206224+0000) 2024-02-20T09:03:46.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:46.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:46.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:46.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:46.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:46.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:46.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:46.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:46.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:46.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:46.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:46.468+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:46.468+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:46.468+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:46.469+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:46.513+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:46.514286+0000) 2024-02-20T09:03:47.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:47.013+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:47.014396+0000) 2024-02-20T09:03:47.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:47.207+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:47.208676+0000) 2024-02-20T09:03:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:47.570+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:47.570+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:47.570+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:47.571+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:47.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:47.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:47.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:47.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:47.485+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:47.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:47.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:47.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:47.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:47.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:47.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:47.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:47.515+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:47.516614+0000) 2024-02-20T09:03:48.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:48.015+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:48.016790+0000) 2024-02-20T09:03:48.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:48.210+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:48.211150+0000) 2024-02-20T09:03:48.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:48.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:48.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:48.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:48.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:48.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:48.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:48.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:48.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:48.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:48.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:48.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:48.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:48.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:48.475+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:48.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:48.518+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:48.519047+0000) 2024-02-20T09:03:49.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:49.017+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.018707+0000) 2024-02-20T09:03:49.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:49.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.212980+0000) 2024-02-20T09:03:49.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:49.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:49.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:49.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:49.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:49.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:49.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:49.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:49.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:49.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:49.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:49.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:49.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:49.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:49.455+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:49.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:49.519+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.520756+0000) 2024-02-20T09:03:50.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:50.017+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.018933+0000) 2024-02-20T09:03:50.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:50.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.213247+0000) 2024-02-20T09:03:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:50.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:50.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:50.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:50.413+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:50.520+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.521030+0000) 2024-02-20T09:03:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:50.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:50.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:50.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:50.486+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:50.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:50.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:50.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:50.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:51.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:50.859+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.860208+0000) 2024-02-20T09:03:51.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:50.857+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.858912+0000) 2024-02-20T09:03:51.155 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:50.859+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54268 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:51.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:50.857+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.858666+0000) 2024-02-20T09:03:51.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:51.018+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:51.019116+0000) 2024-02-20T09:03:51.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:51.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:51.213423+0000) 2024-02-20T09:03:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:51.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:51.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:51.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:51.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:51.520+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:51.521278+0000) 2024-02-20T09:03:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:51.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:51.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:51.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:51.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:51.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:51.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:51.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:51.624+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:52.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:52.020+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:52.021847+0000) 2024-02-20T09:03:52.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:52.215+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:52.216894+0000) 2024-02-20T09:03:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:52.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:52.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:52.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:52.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:52.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:52.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:52.611+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:52.612+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:52.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:52.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:52.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:52.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:52.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:52.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:52.523+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:52.524839+0000) 2024-02-20T09:03:53.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:53.024+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:53.025153+0000) 2024-02-20T09:03:53.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:53.218+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:53.219667+0000) 2024-02-20T09:03:53.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:53.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:53.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:53.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:53.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:53.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:53.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:53.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:53.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:53.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:53.526+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:53.527092+0000) 2024-02-20T09:03:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:53.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:53.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:53.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:53.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:54.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:54.221+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.221956+0000) 2024-02-20T09:03:54.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:54.026+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.027657+0000) 2024-02-20T09:03:54.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:54.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:54.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:54.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:54.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:54.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:54.528+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.529255+0000) 2024-02-20T09:03:54.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:54.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:54.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:54.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:54.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:54.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:54.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:54.496+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:54.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:54.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:54.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:54.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:55.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:55.221+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.222125+0000) 2024-02-20T09:03:55.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:55.026+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.027842+0000) 2024-02-20T09:03:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:55.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:55.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:55.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:55.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:55.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:55.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:55.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:55.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:55.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:55.528+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.529441+0000) 2024-02-20T09:03:56.029 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:03:55.866+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54307 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:03:56.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:55.864+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.865617+0000) 2024-02-20T09:03:56.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:55.864+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.865153+0000) 2024-02-20T09:03:56.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:55.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:56.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:55.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:56.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:55.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:56.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:55.751+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:56.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:55.863+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.864718+0000) 2024-02-20T09:03:56.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:56.221+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:56.222310+0000) 2024-02-20T09:03:56.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:56.027+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:56.028014+0000) 2024-02-20T09:03:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:56.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:56.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:56.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:56.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:56.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:56.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:56.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:56.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:56.528+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:56.529592+0000) 2024-02-20T09:03:57.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:56.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:57.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:56.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:57.032 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:56.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:57.032 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:56.740+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:57.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:57.224+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:57.224950+0000) 2024-02-20T09:03:57.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:57.029+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:57.030351+0000) 2024-02-20T09:03:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:57.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:57.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:57.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:57.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:57.532+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:57.532962+0000) 2024-02-20T09:03:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:57.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:57.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:57.413+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:57.414+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:58.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:57.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:58.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:57.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:58.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:57.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:58.035 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:57.765+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:58.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:58.227+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:58.228309+0000) 2024-02-20T09:03:58.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:58.032+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:58.033669+0000) 2024-02-20T09:03:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:58.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:58.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:58.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:58.419+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:58.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:58.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:58.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:58.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:58.534+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:58.535312+0000) 2024-02-20T09:03:59.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:58.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:59.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:58.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:59.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:58.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:03:59.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:58.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:59.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:59.229+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.229951+0000) 2024-02-20T09:03:59.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:59.035+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.036648+0000) 2024-02-20T09:03:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:59.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:59.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:59.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:03:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:03:59.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:59.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:59.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:59.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:03:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:59.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:03:59.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:03:59.536+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.536965+0000) 2024-02-20T09:04:00.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:59.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:00.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:59.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:00.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:59.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:00.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:03:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:03:59.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:00.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:00.035+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.036821+0000) 2024-02-20T09:04:00.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:00.229+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.230140+0000) 2024-02-20T09:04:00.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:00.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:00.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:00.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:00.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:00.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:00.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:00.536+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.537144+0000) 2024-02-20T09:04:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:00.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:00.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:00.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:00.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:00.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:00.873+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54346 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:00.870+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.871727+0000) 2024-02-20T09:04:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:00.870+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.871049+0000) 2024-02-20T09:04:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:00.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:00.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:00.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:00.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:00.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:00.871+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.872243+0000) 2024-02-20T09:04:01.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:01.229+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:01.230315+0000) 2024-02-20T09:04:01.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:01.036+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:01.036955+0000) 2024-02-20T09:04:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:01.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:01.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:01.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:01.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:01.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:01.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:01.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:01.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:01.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:01.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:01.536+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:01.537311+0000) 2024-02-20T09:04:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:01.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:01.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:01.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:01.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:02.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:02.039+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:02.040358+0000) 2024-02-20T09:04:02.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:02.232+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:02.233088+0000) 2024-02-20T09:04:02.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:02.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:02.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:02.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:02.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:02.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:02.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:02.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:02.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:02.538+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:02.539779+0000) 2024-02-20T09:04:02.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:02.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:02.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:02.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:02.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:02.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:02.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:02.455+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:02.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:02.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:02.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:02.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:03.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:03.235+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:03.236424+0000) 2024-02-20T09:04:03.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:03.042+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:03.043076+0000) 2024-02-20T09:04:03.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:03.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:03.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:03.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:03.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:03.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:03.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:03.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:03.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:03.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:03.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:03.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:03.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:03.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:03.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:03.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:03.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:03.541+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:03.542337+0000) 2024-02-20T09:04:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:03.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:03.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:03.700+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:03.701+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:04.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:04.237+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.238111+0000) 2024-02-20T09:04:04.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:04.043+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.044584+0000) 2024-02-20T09:04:04.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:04.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:04.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:04.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:04.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:04.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:04.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:04.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:04.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:04.543+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.544147+0000) 2024-02-20T09:04:04.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:04.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:04.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:04.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:04.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:04.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:04.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:04.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:04.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:04.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:04.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:04.708+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:05.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:05.237+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.238325+0000) 2024-02-20T09:04:05.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:05.043+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.044770+0000) 2024-02-20T09:04:05.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:05.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:05.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:05.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:05.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:05.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:05.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:05.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:05.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:05.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:05.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:05.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:05.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:05.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:05.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:05.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:05.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:05.543+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.544341+0000) 2024-02-20T09:04:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:05.877+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.878125+0000) 2024-02-20T09:04:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:05.877+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.878606+0000) 2024-02-20T09:04:05.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:05.880+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54385 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:05.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:05.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:05.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:05.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:05.877+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.878789+0000) 2024-02-20T09:04:06.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:06.237+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:06.238516+0000) 2024-02-20T09:04:06.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:06.043+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:06.044918+0000) 2024-02-20T09:04:06.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:06.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:06.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:06.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:06.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:06.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:06.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:06.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:06.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:06.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:06.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:06.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:06.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:06.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:06.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:06.530+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:06.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:06.543+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:06.544523+0000) 2024-02-20T09:04:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:06.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:06.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:06.694+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:06.695+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:07.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:07.240+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:07.240969+0000) 2024-02-20T09:04:07.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:07.046+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:07.047363+0000) 2024-02-20T09:04:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:07.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:07.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:07.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:07.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:07.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:07.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:07.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:07.526+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:07.546+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:07.547044+0000) 2024-02-20T09:04:08.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:07.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:08.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:07.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:08.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:07.730+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:08.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:07.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:08.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:08.243+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:08.244524+0000) 2024-02-20T09:04:08.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:08.049+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:08.050027+0000) 2024-02-20T09:04:08.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:08.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:08.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:08.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:08.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:08.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:08.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:08.548+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:08.549506+0000) 2024-02-20T09:04:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:08.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:08.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:08.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:08.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:08.496+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:08.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:08.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:08.716+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:08.717+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:09.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:09.245+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.246130+0000) 2024-02-20T09:04:09.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:09.052+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.053262+0000) 2024-02-20T09:04:09.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:09.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:09.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:09.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:09.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:09.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:09.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:09.502+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:09.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:09.550+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.551363+0000) 2024-02-20T09:04:09.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:09.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:09.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:09.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:09.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:09.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:09.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:09.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:09.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:09.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:09.707+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:09.708+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:10.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:10.245+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.246306+0000) 2024-02-20T09:04:10.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:10.052+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.053430+0000) 2024-02-20T09:04:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:10.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:10.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:10.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:10.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:10.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:10.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:10.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:10.538+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:10.550+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.551523+0000) 2024-02-20T09:04:11.010 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:10.887+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54424 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:11.010 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:10.884+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.885610+0000) 2024-02-20T09:04:11.010 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:10.885+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.886586+0000) 2024-02-20T09:04:11.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:10.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:11.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:10.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:11.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:10.756+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:11.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:10.757+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:11.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:10.886+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.887246+0000) 2024-02-20T09:04:11.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:11.245+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:11.246499+0000) 2024-02-20T09:04:11.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:11.052+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:11.053597+0000) 2024-02-20T09:04:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:11.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:11.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:11.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:11.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:11.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:11.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:11.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:11.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:11.550+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:11.551690+0000) 2024-02-20T09:04:12.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:11.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:12.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:11.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:12.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:11.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:12.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:11.748+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:12.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:12.246+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:12.248536+0000) 2024-02-20T09:04:12.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:12.053+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:12.055692+0000) 2024-02-20T09:04:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:12.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:12.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:12.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:12.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:12.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:12.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:12.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:12.514+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:12.553+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:12.555211+0000) 2024-02-20T09:04:12.991 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:12.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:12.992 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:12.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:12.992 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:12.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:12.992 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:12.736+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:13.252 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:13.057+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:13.059229+0000) 2024-02-20T09:04:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:13.248+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:13.250798+0000) 2024-02-20T09:04:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:13.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:13.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:13.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:13.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:13.781 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:13.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:13.781 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:13.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:13.781 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:13.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:13.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:13.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:13.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:13.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:13.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:13.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:13.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:13.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:13.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:13.521+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:13.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:13.556+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:13.558525+0000) 2024-02-20T09:04:14.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:14.059+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.061246+0000) 2024-02-20T09:04:14.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:14.250+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.252515+0000) 2024-02-20T09:04:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:14.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:14.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:14.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:14.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:14.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:14.558+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.559971+0000) 2024-02-20T09:04:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:14.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:14.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:14.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:14.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:14.563+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:15.062 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:14.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:15.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:14.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:15.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:14.743+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:15.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:14.744+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:15.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:15.251+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.252697+0000) 2024-02-20T09:04:15.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:15.060+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.061414+0000) 2024-02-20T09:04:15.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:15.559+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.560067+0000) 2024-02-20T09:04:15.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:15.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:15.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:15.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:15.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:15.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:15.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:15.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:15.891 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:15.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:15.891 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:15.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:15.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:15.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:15.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:15.597+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:15.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:15.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:15.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:15.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:15.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:15.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:15.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:15.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:16.171 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:15.895+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54463 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:16.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:15.894+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.895352+0000) 2024-02-20T09:04:16.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:15.893+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.893968+0000) 2024-02-20T09:04:16.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:15.894+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.895511+0000) 2024-02-20T09:04:16.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:16.060+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:16.061685+0000) 2024-02-20T09:04:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:16.251+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:16.252865+0000) 2024-02-20T09:04:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:16.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:16.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:16.775+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:16.776+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:16.559+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:16.560265+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:16.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:16.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:16.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:16.601+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:16.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:16.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:16.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:16.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:17.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:17.254+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:17.255568+0000) 2024-02-20T09:04:17.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:17.063+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:17.064363+0000) 2024-02-20T09:04:17.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:17.561+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:17.562745+0000) 2024-02-20T09:04:17.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:17.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:17.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:17.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:17.591+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:17.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:17.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:17.773+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:17.774+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:17.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:17.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:17.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:17.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:17.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:18.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:18.258+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:18.258969+0000) 2024-02-20T09:04:18.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:18.066+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:18.067099+0000) 2024-02-20T09:04:18.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:18.564+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:18.565013+0000) 2024-02-20T09:04:18.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:18.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:18.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:18.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:18.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:18.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:18.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:18.551+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:18.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:18.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:18.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:18.630+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:18.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:18.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:18.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:18.735+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:19.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:19.068+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.069748+0000) 2024-02-20T09:04:19.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:19.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.261134+0000) 2024-02-20T09:04:19.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:19.566+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.567206+0000) 2024-02-20T09:04:19.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:19.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:19.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:19.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:19.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:19.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:19.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:19.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:19.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:19.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:19.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:19.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:19.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:19.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:19.605+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:19.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:19.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:19.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:19.711+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:20.335 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:20.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.261314+0000) 2024-02-20T09:04:20.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:20.069+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.069929+0000) 2024-02-20T09:04:20.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:20.566+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.567401+0000) 2024-02-20T09:04:20.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:20.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:20.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:20.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:20.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:20.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:20.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:20.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:20.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:20.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:20.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:20.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:20.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:20.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:20.900 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:20.593+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:20.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:20.726+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:20.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:20.726+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:20.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:20.726+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:20.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:20.727+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:21.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:20.899+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.900470+0000) 2024-02-20T09:04:21.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:20.900+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.901707+0000) 2024-02-20T09:04:21.176 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:20.902+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54502 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:21.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:20.900+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.901325+0000) 2024-02-20T09:04:21.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:21.069+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:21.070129+0000) 2024-02-20T09:04:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:21.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:21.261495+0000) 2024-02-20T09:04:21.881 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:21.566+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:21.567577+0000) 2024-02-20T09:04:21.881 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:21.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:21.881 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:21.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:21.882 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:21.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:21.882 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:21.585+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:21.882 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:21.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:21.882 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:21.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:21.882 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:21.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:21.882 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:21.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:21.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:21.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:21.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:21.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:21.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:21.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:21.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:21.687+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:22.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:22.071+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:22.072591+0000) 2024-02-20T09:04:22.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:22.263+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:22.264040+0000) 2024-02-20T09:04:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:22.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:22.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:22.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:22.563+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:22.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:22.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:22.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:22.647+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:22.569+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:22.570259+0000) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:22.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:22.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:22.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:22.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:22.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:22.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:22.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:23.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:23.265+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:23.266461+0000) 2024-02-20T09:04:23.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:23.074+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:23.075057+0000) 2024-02-20T09:04:23.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:23.571+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:23.572428+0000) 2024-02-20T09:04:23.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:23.593+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:23.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:23.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:23.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:23.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:23.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:23.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:23.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:23.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:23.593+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:23.593+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:23.593+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:23.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:23.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:23.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:23.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:24.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:24.267+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.268493+0000) 2024-02-20T09:04:24.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:24.075+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.076875+0000) 2024-02-20T09:04:24.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:24.574+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.575121+0000) 2024-02-20T09:04:24.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:24.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:24.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:24.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:24.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:24.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:24.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:24.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:24.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:24.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:24.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:24.513+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:24.609+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:24.611+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:24.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:24.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:24.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:24.646+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:25.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:25.267+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.268763+0000) 2024-02-20T09:04:25.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:25.076+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.077193+0000) 2024-02-20T09:04:25.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:25.574+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.575343+0000) 2024-02-20T09:04:25.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:25.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:25.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:25.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:25.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:25.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:25.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:25.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:25.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:25.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:25.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:25.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:25.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:25.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:25.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:25.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:25.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:25.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:25.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:25.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:25.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:25.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:25.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:25.597+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:26.178 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:25.909+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54540 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:26.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:25.908+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.909020+0000) 2024-02-20T09:04:26.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:25.906+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.907360+0000) 2024-02-20T09:04:26.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:25.908+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.909417+0000) 2024-02-20T09:04:26.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:26.076+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:26.077439+0000) 2024-02-20T09:04:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:26.268+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:26.268951+0000) 2024-02-20T09:04:26.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:26.574+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:26.575642+0000) 2024-02-20T09:04:26.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:26.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:26.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:26.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:26.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:26.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:26.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:26.597+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:26.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:26.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:26.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:26.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:26.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:26.590+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:26.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:26.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:26.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:26.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:26.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:26.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:26.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:26.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:26.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:26.493+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:27.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:27.271+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:27.272451+0000) 2024-02-20T09:04:27.344 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:27.078+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:27.079906+0000) 2024-02-20T09:04:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:27.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:27.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:27.581+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:27.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:27.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:27.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:27.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:27.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:27.577+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:27.578523+0000) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:27.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:27.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:27.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:27.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:27.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:27.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:28.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:28.274+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:28.275597+0000) 2024-02-20T09:04:28.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:28.081+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:28.082409+0000) 2024-02-20T09:04:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:28.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:28.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:28.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:28.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:28.580+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:28.581229+0000) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:28.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:28.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:28.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:28.582+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:28.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:28.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:28.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:28.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:28.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:28.557+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:29.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:29.277+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.278154+0000) 2024-02-20T09:04:29.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:29.083+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.084520+0000) 2024-02-20T09:04:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:29.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:29.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:29.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:29.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:29.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:29.528+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:29.528+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:29.528+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:29.529+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:29.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:29.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:29.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:29.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:29.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:29.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:29.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:29.551+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:29.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:29.582+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.583097+0000) 2024-02-20T09:04:30.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:30.277+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.278333+0000) 2024-02-20T09:04:30.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:30.083+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.084677+0000) 2024-02-20T09:04:30.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:30.582+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.583278+0000) 2024-02-20T09:04:30.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:30.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:30.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:30.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:30.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:30.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:30.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:30.421+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:30.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:30.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:30.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:30.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:30.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:30.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:30.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:30.510+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:30.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:30.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:30.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:30.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:30.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:30.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:30.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:30.596+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:31.181 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:30.917+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54580 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:31.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:30.914+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.915261+0000) 2024-02-20T09:04:31.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:30.914+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.915590+0000) 2024-02-20T09:04:31.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:30.914+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.915303+0000) 2024-02-20T09:04:31.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:31.083+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:31.084873+0000) 2024-02-20T09:04:31.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:31.277+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:31.278499+0000) 2024-02-20T09:04:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:31.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:31.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:31.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:31.490+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:31.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:31.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:31.548+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:31.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:31.582+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:31.583390+0000) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:31.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:31.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:31.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:31.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:31.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:32.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:32.279+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:32.280865+0000) 2024-02-20T09:04:32.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:32.086+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:32.087081+0000) 2024-02-20T09:04:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:32.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:32.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:32.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:32.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:32.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:32.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:32.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:32.521+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:32.584+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:32.585519+0000) 2024-02-20T09:04:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:32.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:32.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:32.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:32.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:32.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:33.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:33.283+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:33.284462+0000) 2024-02-20T09:04:33.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:33.088+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:33.089704+0000) 2024-02-20T09:04:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:33.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:33.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:33.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:33.554+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:33.588+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:33.589043+0000) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:33.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:33.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:33.524+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:33.526+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:33.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:33.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:33.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:33.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:33.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:33.435+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:34.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:34.091+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:34.092418+0000) 2024-02-20T09:04:34.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:34.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:34.287050+0000) 2024-02-20T09:04:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:34.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:34.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:34.564+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:34.565+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:34.590+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:34.591673+0000) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:34.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:34.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:34.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:34.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:34.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:34.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:34.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:34.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:34.480+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:35.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:35.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.287244+0000) 2024-02-20T09:04:35.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:35.091+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.092591+0000) 2024-02-20T09:04:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:35.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:35.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:35.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:35.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:35.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:35.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:35.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:35.445+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:35.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:35.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:35.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:35.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:35.536+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:35.590+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.591851+0000) 2024-02-20T09:04:36.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:35.924+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54618 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:36.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:35.923+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.924608+0000) 2024-02-20T09:04:36.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:35.923+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.924799+0000) 2024-02-20T09:04:36.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:35.922+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.923630+0000) 2024-02-20T09:04:36.308 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:36.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:36.287453+0000) 2024-02-20T09:04:36.308 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:36.091+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:36.092790+0000) 2024-02-20T09:04:36.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:36.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:36.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:36.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:36.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:36.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:36.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:36.540+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:36.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:36.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:36.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:36.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:36.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:36.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:36.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:36.519+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:36.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:36.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:36.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:36.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:36.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:36.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:36.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:36.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:36.591+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:36.592033+0000) 2024-02-20T09:04:37.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:37.288+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:37.289357+0000) 2024-02-20T09:04:37.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:37.094+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:37.095267+0000) 2024-02-20T09:04:37.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:37.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:37.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:37.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:37.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:37.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:37.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:37.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:37.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:37.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:37.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:37.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:37.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:37.584+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:37.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:37.593+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:37.594514+0000) 2024-02-20T09:04:38.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:38.291+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:38.291950+0000) 2024-02-20T09:04:38.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:38.096+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:38.097285+0000) 2024-02-20T09:04:38.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:38.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:38.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:38.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:38.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:38.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:38.408+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:38.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:38.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:38.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:38.498+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:38.595+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:38.596543+0000) 2024-02-20T09:04:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:38.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:38.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:38.632+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:38.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:39.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:39.292+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:39.293793+0000) 2024-02-20T09:04:39.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:39.098+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:39.099800+0000) 2024-02-20T09:04:39.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:39.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:39.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:39.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:39.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:39.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:39.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:39.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:39.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:39.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:39.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:39.598+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:39.599171+0000) 2024-02-20T09:04:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:39.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:39.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:39.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:39.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:39.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:40.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:40.293+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.293958+0000) 2024-02-20T09:04:40.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:40.099+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.099951+0000) 2024-02-20T09:04:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:40.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:40.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:40.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:40.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:40.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:40.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:40.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:40.407+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:40.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:40.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:40.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:40.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:40.598+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.599348+0000) 2024-02-20T09:04:41.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:40.931+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54658 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:41.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:40.929+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.930213+0000) 2024-02-20T09:04:41.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:40.929+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.930437+0000) 2024-02-20T09:04:41.054 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:40.930+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.931385+0000) 2024-02-20T09:04:41.308 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:41.293+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:41.294120+0000) 2024-02-20T09:04:41.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:41.099+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:41.100137+0000) 2024-02-20T09:04:41.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:41.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:41.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:41.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:41.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:41.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:41.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:41.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:41.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:41.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:41.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:41.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:41.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:41.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:41.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:41.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:41.882 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:41.598+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:41.599526+0000) 2024-02-20T09:04:41.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:41.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:41.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:41.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:41.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:41.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:41.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:41.612+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:42.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:42.101+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:42.102504+0000) 2024-02-20T09:04:42.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:42.295+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:42.296628+0000) 2024-02-20T09:04:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:42.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:42.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:42.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:42.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:42.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:42.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:42.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:42.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:42.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:42.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:42.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:42.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:42.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:42.600+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:42.601554+0000) 2024-02-20T09:04:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:42.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:42.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:42.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:42.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:43.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:43.299+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:43.300178+0000) 2024-02-20T09:04:43.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:43.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:43.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:43.103+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:43.104543+0000) 2024-02-20T09:04:43.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:43.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:43.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:43.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:43.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:43.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:43.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:43.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:43.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:43.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:43.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:43.603+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:43.604148+0000) 2024-02-20T09:04:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:43.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:43.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:43.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:43.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:44.301+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:44.302833+0000) 2024-02-20T09:04:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:44.106+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:44.107192+0000) 2024-02-20T09:04:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:44.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:44.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:44.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:44.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:44.294+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:44.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:44.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:44.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:44.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:44.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:44.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:44.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:44.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:44.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:44.577+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:44.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:44.604+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:44.605757+0000) 2024-02-20T09:04:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:45.302+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.303023+0000) 2024-02-20T09:04:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:45.106+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.107373+0000) 2024-02-20T09:04:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:45.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:45.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:45.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:45.294+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:45.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:45.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:45.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:45.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:45.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:45.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:45.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:45.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:45.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:45.605+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.605961+0000) 2024-02-20T09:04:45.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:45.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:45.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:45.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:45.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:45.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:45.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:45.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:46.059 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:45.937+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54696 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:46.059 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:45.936+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.937275+0000) 2024-02-20T09:04:46.060 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:45.936+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.937587+0000) 2024-02-20T09:04:46.060 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:45.935+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.936575+0000) 2024-02-20T09:04:46.309 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:46.302+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:46.303199+0000) 2024-02-20T09:04:46.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:46.106+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:46.107573+0000) 2024-02-20T09:04:46.309 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:46.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:46.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:46.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:46.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:46.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:46.310 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:46.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:46.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:46.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:46.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:46.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:46.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:46.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:46.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:46.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:46.605+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:46.606056+0000) 2024-02-20T09:04:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:46.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:46.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:46.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:46.658+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:47.304+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:47.305772+0000) 2024-02-20T09:04:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:47.110+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:47.111208+0000) 2024-02-20T09:04:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:47.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:47.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:47.251+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:47.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:47.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:47.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:47.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:47.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:47.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:47.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:47.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:47.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:47.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:47.607+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:47.608703+0000) 2024-02-20T09:04:47.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:47.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:47.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:47.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:47.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:47.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:47.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:47.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:48.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:48.307+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:48.308052+0000) 2024-02-20T09:04:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:48.113+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:48.114550+0000) 2024-02-20T09:04:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:48.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:48.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:48.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:48.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:48.610+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:48.610954+0000) 2024-02-20T09:04:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:48.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:48.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:48.630+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:48.631+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:48.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:48.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:48.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:48.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:48.587+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:49.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:49.309+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:49.310057+0000) 2024-02-20T09:04:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:49.116+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:49.117214+0000) 2024-02-20T09:04:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:49.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:49.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:49.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:49.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:49.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:49.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:49.611+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:49.612725+0000) 2024-02-20T09:04:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:49.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:49.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:49.666+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:49.667+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:49.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:49.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:49.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:49.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:49.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:50.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:50.309+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.310231+0000) 2024-02-20T09:04:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:50.116+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.117411+0000) 2024-02-20T09:04:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:50.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:50.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:50.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:50.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:50.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:50.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:50.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:50.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:50.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:50.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:50.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:50.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:50.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:50.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:50.611+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.612905+0000) 2024-02-20T09:04:50.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:50.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:50.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:50.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:50.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:50.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:50.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:50.713+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:51.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:50.944+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.945114+0000) 2024-02-20T09:04:51.067 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:50.945+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54736 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:51.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:50.942+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.943068+0000) 2024-02-20T09:04:51.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:50.942+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.943539+0000) 2024-02-20T09:04:51.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:51.309+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:51.310429+0000) 2024-02-20T09:04:51.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:51.116+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:51.117613+0000) 2024-02-20T09:04:51.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:51.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:51.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:51.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:51.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:51.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:51.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:51.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:51.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:51.612+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:51.613082+0000) 2024-02-20T09:04:51.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:51.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:51.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:51.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:51.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:51.670+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:51.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:51.671+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:51.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:51.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:51.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:51.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:51.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:51.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:51.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:51.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:52.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:52.118+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:52.119833+0000) 2024-02-20T09:04:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:52.311+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:52.312829+0000) 2024-02-20T09:04:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:52.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:52.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:52.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:52.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:52.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:52.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:52.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:52.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:52.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:52.664+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:52.928 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:52.665+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:52.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:52.615+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:52.616627+0000) 2024-02-20T09:04:52.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:52.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:52.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:52.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:52.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:52.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:52.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:52.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:53.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:53.122+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:53.123264+0000) 2024-02-20T09:04:53.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:53.314+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:53.315608+0000) 2024-02-20T09:04:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:53.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:53.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:53.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:53.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:53.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:53.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:53.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:53.639+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:53.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:53.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:53.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:53.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:53.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:53.618+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:53.619093+0000) 2024-02-20T09:04:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:54.316+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:54.317433+0000) 2024-02-20T09:04:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:54.124+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:54.125176+0000) 2024-02-20T09:04:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:54.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:54.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:54.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:54.230+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:54.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:54.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:54.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:54.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:54.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:54.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:54.601+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:54.602+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:54.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:54.620+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:54.621025+0000) 2024-02-20T09:04:55.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:55.316+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.317622+0000) 2024-02-20T09:04:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:55.124+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.125355+0000) 2024-02-20T09:04:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:55.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:55.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:55.239+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:55.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:55.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:55.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:55.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:55.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:55.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:55.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:55.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:55.582+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:55.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:55.620+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.621205+0000) 2024-02-20T09:04:55.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:55.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:55.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:55.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:55.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:55.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:55.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:55.586+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:56.214 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:04:55.953+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54774 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:04:56.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:55.949+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.950035+0000) 2024-02-20T09:04:56.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:55.950+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.951003+0000) 2024-02-20T09:04:56.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:55.950+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.951782+0000) 2024-02-20T09:04:56.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:56.124+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:56.125540+0000) 2024-02-20T09:04:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:56.316+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:56.317804+0000) 2024-02-20T09:04:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:56.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:56.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:56.256+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:56.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:56.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:56.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:56.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:56.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:56.563+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:56.620+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:56.621386+0000) 2024-02-20T09:04:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:56.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:56.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:56.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:56.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:56.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:57.319+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:57.320229+0000) 2024-02-20T09:04:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:57.127+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:57.128274+0000) 2024-02-20T09:04:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:57.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:57.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:57.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:57.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:57.292+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:57.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:57.622+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:57.623918+0000) 2024-02-20T09:04:57.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:57.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:57.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:57.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:57.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:57.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:57.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:57.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:57.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:57.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:57.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:57.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:57.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:58.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:58.322+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:58.323838+0000) 2024-02-20T09:04:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:58.129+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:58.130833+0000) 2024-02-20T09:04:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:58.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:58.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:58.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:58.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:58.288+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:58.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:58.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:58.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:58.611+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:58.625+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:58.626557+0000) 2024-02-20T09:04:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:58.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:58.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:58.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:58.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:58.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:59.325+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:59.326592+0000) 2024-02-20T09:04:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:59.131+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:59.132543+0000) 2024-02-20T09:04:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:59.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:59.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:59.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:04:59.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:04:59.307+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:59.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:59.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:59.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:04:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:04:59.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:59.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:59.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:59.600+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:04:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:59.601+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:04:59.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:04:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:04:59.627+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:59.628271+0000) 2024-02-20T09:05:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:00.325+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.326784+0000) 2024-02-20T09:05:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:00.131+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.132729+0000) 2024-02-20T09:05:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:00.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:00.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:00.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:00.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:00.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:00.627+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.628723+0000) 2024-02-20T09:05:00.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:00.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:00.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:00.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:00.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:00.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:00.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:00.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:00.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:00.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:00.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:00.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:00.581+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:01.214 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:00.960+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54814 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:01.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:00.957+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.958610+0000) 2024-02-20T09:05:01.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:00.959+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.960024+0000) 2024-02-20T09:05:01.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:00.957+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.958344+0000) 2024-02-20T09:05:01.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:01.132+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:01.132940+0000) 2024-02-20T09:05:01.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:01.326+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:01.326982+0000) 2024-02-20T09:05:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:01.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:01.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:01.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:01.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:01.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:01.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:01.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:01.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:01.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:01.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:01.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:01.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:01.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:01.627+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:01.628823+0000) 2024-02-20T09:05:01.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:01.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:01.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:01.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:01.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:01.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:01.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:01.636+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:02.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:02.328+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:02.329488+0000) 2024-02-20T09:05:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:02.134+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:02.134930+0000) 2024-02-20T09:05:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:02.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:02.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:02.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:02.318+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:02.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:02.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:02.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:02.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:02.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:02.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:02.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:02.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:02.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:02.630+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:02.631299+0000) 2024-02-20T09:05:02.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:02.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:02.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:02.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:02.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:02.639+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:02.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:02.640+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:03.332+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:03.333052+0000) 2024-02-20T09:05:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:03.136+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:03.137497+0000) 2024-02-20T09:05:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:03.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:03.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:03.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:03.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:03.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:03.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:03.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:03.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:03.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:03.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:03.632+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:03.633372+0000) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:03.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:03.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:03.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:03.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:03.689+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:04.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:04.334862+0000) 2024-02-20T09:05:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:04.138+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:04.139626+0000) 2024-02-20T09:05:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:04.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:04.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:04.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:04.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:04.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:04.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:04.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:04.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:04.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:04.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:04.634+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:04.635709+0000) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:04.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:04.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:04.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:04.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:04.659+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:05.407 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:05.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.335054+0000) 2024-02-20T09:05:05.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:05.137+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.139811+0000) 2024-02-20T09:05:05.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:05.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:05.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:05.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:05.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:05.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:05.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:05.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:05.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:05.633+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.635876+0000) 2024-02-20T09:05:05.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:05.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:05.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:05.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:05.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:05.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:05.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:05.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:05.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:05.474+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:05.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:05.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:05.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:05.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:05.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:05.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:06.215 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:05.966+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54852 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:06.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:05.963+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.965407+0000) 2024-02-20T09:05:06.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:05.965+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.967520+0000) 2024-02-20T09:05:06.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:05.963+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.965846+0000) 2024-02-20T09:05:06.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:06.138+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:06.139975+0000) 2024-02-20T09:05:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:06.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:06.335221+0000) 2024-02-20T09:05:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:06.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:06.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:06.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:06.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:06.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:06.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:06.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:06.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:06.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:06.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:06.634+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:06.636051+0000) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:06.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:06.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:06.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:06.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:06.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:07.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:07.336+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:07.337576+0000) 2024-02-20T09:05:07.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:07.142+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:07.143157+0000) 2024-02-20T09:05:07.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:07.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:07.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:07.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:07.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:07.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:07.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:07.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:07.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:07.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:07.636+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:07.638488+0000) 2024-02-20T09:05:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:07.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:07.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:07.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:07.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:07.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:07.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:08.339+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:08.340182+0000) 2024-02-20T09:05:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:08.144+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:08.145570+0000) 2024-02-20T09:05:08.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:08.640+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:08.640966+0000) 2024-02-20T09:05:08.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:08.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:08.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:08.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:08.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:08.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:08.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:08.673+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:08.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:08.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:08.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:08.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:08.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:08.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:08.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:08.514+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:08.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:08.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:08.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:08.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:08.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:08.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:08.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:08.472+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:09.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:09.341+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:09.342205+0000) 2024-02-20T09:05:09.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:09.146+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:09.147338+0000) 2024-02-20T09:05:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:09.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:09.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:09.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:09.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:09.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:09.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:09.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:09.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:09.642+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:09.643015+0000) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:09.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:09.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:09.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:09.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:09.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:10.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:10.341+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.342363+0000) 2024-02-20T09:05:10.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:10.146+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.147529+0000) 2024-02-20T09:05:10.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:10.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:10.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:10.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:10.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:10.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:10.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:10.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:10.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:10.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:10.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:10.642+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.643204+0000) 2024-02-20T09:05:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:10.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:10.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:10.672+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:10.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:10.673+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:10.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:10.446+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:11.098 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:10.974+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54892 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:11.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:10.971+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.972572+0000) 2024-02-20T09:05:11.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:10.972+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.973009+0000) 2024-02-20T09:05:11.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:10.972+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.973139+0000) 2024-02-20T09:05:11.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:11.341+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:11.342543+0000) 2024-02-20T09:05:11.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:11.146+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:11.147699+0000) 2024-02-20T09:05:11.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:11.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:11.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:11.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:11.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:11.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:11.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:11.588+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:11.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:11.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:11.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:11.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:11.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:11.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:11.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:11.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:11.642+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:11.643309+0000) 2024-02-20T09:05:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:11.704+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:11.704+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:11.704+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:11.705+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:12.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:12.344+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:12.345119+0000) 2024-02-20T09:05:12.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:12.150+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:12.151259+0000) 2024-02-20T09:05:12.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:12.400+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:12.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:12.644+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:12.645861+0000) 2024-02-20T09:05:12.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:12.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:12.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:12.400+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:12.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:12.400+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:12.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:12.401+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:12.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:12.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:12.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:12.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:12.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:12.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:12.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:12.545+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:12.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:12.692+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:12.693+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:13.425 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:13.347+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:13.348417+0000) 2024-02-20T09:05:13.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:13.152+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:13.153699+0000) 2024-02-20T09:05:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:13.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:13.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:13.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:13.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:13.648+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:13.649342+0000) 2024-02-20T09:05:13.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:13.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:13.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:13.657+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:13.658+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:13.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:13.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:13.423+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:13.424+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:14.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:14.349+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:14.350358+0000) 2024-02-20T09:05:14.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:14.154+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:14.155739+0000) 2024-02-20T09:05:14.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:14.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:14.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:14.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:14.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:14.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:14.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:14.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:14.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:14.651+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:14.652019+0000) 2024-02-20T09:05:14.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:14.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:14.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:14.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:14.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:14.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:14.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:14.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:14.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:14.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:14.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:14.698+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:15.349+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.350592+0000) 2024-02-20T09:05:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:15.154+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.155920+0000) 2024-02-20T09:05:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:15.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:15.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:15.399+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:15.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:15.400+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:15.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:15.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:15.590+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:15.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:15.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:15.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:15.651+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:15.651+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.652202+0000) 2024-02-20T09:05:15.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:15.652+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:16.313 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:15.979+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.980231+0000) 2024-02-20T09:05:16.314 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:15.979+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.980146+0000) 2024-02-20T09:05:16.314 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:15.981+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54930 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:16.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:15.978+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.979526+0000) 2024-02-20T09:05:16.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:16.155+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:16.156101+0000) 2024-02-20T09:05:16.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:16.349+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:16.350779+0000) 2024-02-20T09:05:16.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:16.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:16.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:16.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:16.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:16.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:16.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:16.448+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:16.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:16.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:16.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:16.604+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:16.651+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:16.652391+0000) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:16.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:16.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:16.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:16.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:16.664+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:17.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:17.352+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:17.353208+0000) 2024-02-20T09:05:17.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:17.158+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:17.159569+0000) 2024-02-20T09:05:17.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:17.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:17.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:17.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:17.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:17.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:17.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:17.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:17.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:17.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:17.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:17.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:17.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:17.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:17.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:17.614+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:17.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:17.653+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:17.654832+0000) 2024-02-20T09:05:17.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:17.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:17.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:17.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:17.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:17.712+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:17.713+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:18.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:18.354+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:18.355870+0000) 2024-02-20T09:05:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:18.161+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:18.161951+0000) 2024-02-20T09:05:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:18.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:18.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:18.396+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:18.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:18.397+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:18.656+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:18.657273+0000) 2024-02-20T09:05:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:18.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:18.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:18.710+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:18.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:18.711+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:18.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:18.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:18.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:18.616+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:19.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:19.357+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:19.358232+0000) 2024-02-20T09:05:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:19.163+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:19.164670+0000) 2024-02-20T09:05:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:19.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:19.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:19.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:19.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:19.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:19.384+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:19.658+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:19.659451+0000) 2024-02-20T09:05:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:19.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:19.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:19.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:19.712+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:19.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:19.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:19.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:19.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:19.579+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:20.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:20.357+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.358411+0000) 2024-02-20T09:05:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:20.163+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.164849+0000) 2024-02-20T09:05:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:20.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:20.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:20.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:20.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:20.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:20.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:20.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:20.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:20.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:20.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:20.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:20.534+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:20.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:20.658+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.659638+0000) 2024-02-20T09:05:20.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:20.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:20.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:20.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:20.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:20.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:20.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:20.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:21.109 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:20.988+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 54970 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:21.110 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:20.986+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.987321+0000) 2024-02-20T09:05:21.110 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:20.986+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.987925+0000) 2024-02-20T09:05:21.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:20.986+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.987073+0000) 2024-02-20T09:05:21.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:21.164+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:21.165035+0000) 2024-02-20T09:05:21.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:21.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:21.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:21.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:21.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:21.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:21.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:21.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:21.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:21.357+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:21.358597+0000) 2024-02-20T09:05:21.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:21.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:21.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:21.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:21.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:21.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:21.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:21.526+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:21.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:21.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:21.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:21.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:21.658+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:21.659810+0000) 2024-02-20T09:05:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:22.359+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:22.360589+0000) 2024-02-20T09:05:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:22.167+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:22.168380+0000) 2024-02-20T09:05:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:22.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:22.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:22.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:22.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:22.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:22.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:22.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:22.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:22.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:22.614+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:22.616+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:22.661+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:22.662286+0000) 2024-02-20T09:05:22.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:22.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:22.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:22.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:22.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:22.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:22.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:22.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:23.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:23.363+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:23.364123+0000) 2024-02-20T09:05:23.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:23.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:23.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:23.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:23.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:23.170+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:23.171905+0000) 2024-02-20T09:05:23.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:23.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:23.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:23.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:23.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:23.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:23.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:23.338+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:23.452+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:23.453+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:23.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:23.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:23.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:23.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:23.588+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:23.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:23.664+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:23.665445+0000) 2024-02-20T09:05:24.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:24.365+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:24.366719+0000) 2024-02-20T09:05:24.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:24.173+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:24.173946+0000) 2024-02-20T09:05:24.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:24.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:24.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:24.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:24.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:24.343+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:24.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:24.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:24.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:24.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:24.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:24.430+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:24.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:24.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:24.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:24.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:24.666+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:24.667505+0000) 2024-02-20T09:05:25.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:25.365+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.366903+0000) 2024-02-20T09:05:25.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:25.173+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.174166+0000) 2024-02-20T09:05:25.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:25.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:25.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:25.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:25.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:25.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:25.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:25.358+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:25.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:25.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:25.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:25.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:25.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:25.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:25.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:25.598+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:25.598+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:25.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:25.666+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.667724+0000) 2024-02-20T09:05:26.117 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:25.994+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55008 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:26.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:25.996+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.997017+0000) 2024-02-20T09:05:26.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:25.993+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.994031+0000) 2024-02-20T09:05:26.118 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:25.992+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.993903+0000) 2024-02-20T09:05:26.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:26.173+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:26.174434+0000) 2024-02-20T09:05:26.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:26.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:26.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:26.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:26.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:26.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:26.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:26.576+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:26.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:26.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:26.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:26.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:26.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:26.366+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:26.367090+0000) 2024-02-20T09:05:26.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:26.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:26.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:26.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:26.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:26.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:26.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:26.452+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:26.666+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:26.667901+0000) 2024-02-20T09:05:27.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:27.368+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:27.369570+0000) 2024-02-20T09:05:27.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:27.176+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:27.177608+0000) 2024-02-20T09:05:27.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:27.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:27.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:27.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:27.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:27.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:27.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:27.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:27.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:27.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:27.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:27.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:27.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:27.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:27.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:27.618+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:27.669+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:27.670262+0000) 2024-02-20T09:05:28.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:28.371+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:28.372761+0000) 2024-02-20T09:05:28.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:28.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:28.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:28.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:28.443 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:28.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:28.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:28.179+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:28.180098+0000) 2024-02-20T09:05:28.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:28.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:28.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:28.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:28.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:28.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:28.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:28.388+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:28.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:28.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:28.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:28.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:28.640+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:28.672+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:28.673796+0000) 2024-02-20T09:05:29.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:29.373+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:29.374616+0000) 2024-02-20T09:05:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:29.182+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:29.183112+0000) 2024-02-20T09:05:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:29.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:29.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:29.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:29.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:29.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:29.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:29.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:29.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:29.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:29.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:29.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:29.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:29.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:29.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:29.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:29.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:29.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:29.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:29.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:29.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:29.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:29.674+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:29.675818+0000) 2024-02-20T09:05:30.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:30.373+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.374850+0000) 2024-02-20T09:05:30.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:30.182+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.183296+0000) 2024-02-20T09:05:30.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:30.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:30.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:30.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:30.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:30.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:30.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:30.426+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:30.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:30.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:30.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:30.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:30.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:30.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:30.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:30.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:30.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:30.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:30.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:30.655+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:30.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:30.675+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.675974+0000) 2024-02-20T09:05:31.121 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:31.001+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55049 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:31.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:30.999+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.000626+0000) 2024-02-20T09:05:31.122 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:30.999+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.000000+0000) 2024-02-20T09:05:31.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:30.998+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.999171+0000) 2024-02-20T09:05:31.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:31.182+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.183499+0000) 2024-02-20T09:05:31.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:31.374+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.375063+0000) 2024-02-20T09:05:31.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:31.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:31.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:31.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:31.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:31.438+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:31.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:31.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:31.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:31.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:31.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:31.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:31.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:31.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:31.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:31.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:31.675+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.676174+0000) 2024-02-20T09:05:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:31.704+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:31.704+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:31.704+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:31.705+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:32.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:32.376+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:32.377631+0000) 2024-02-20T09:05:32.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:32.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:32.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:32.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:32.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:32.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:32.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:32.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:32.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:32.186+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:32.187112+0000) 2024-02-20T09:05:32.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:32.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:32.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:32.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:32.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:32.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:32.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:32.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:32.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:32.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:32.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:32.668+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:32.678+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:32.679289+0000) 2024-02-20T09:05:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:33.379+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:33.380089+0000) 2024-02-20T09:05:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:33.188+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:33.189555+0000) 2024-02-20T09:05:33.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:33.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:33.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:33.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:33.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:33.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:33.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:33.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:33.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:33.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:33.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:33.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:33.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:33.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:33.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:33.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:33.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:33.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:33.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:33.655+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:33.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:33.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:33.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:33.681+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:33.682179+0000) 2024-02-20T09:05:34.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:34.381+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:34.381950+0000) 2024-02-20T09:05:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:34.191+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:34.192466+0000) 2024-02-20T09:05:34.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:34.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:34.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:34.406+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:34.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:34.407+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:34.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:34.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:34.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:34.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:34.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:34.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:34.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:34.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:34.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:34.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:34.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:34.623+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:34.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:34.683+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:34.684205+0000) 2024-02-20T09:05:35.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:35.381+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.382127+0000) 2024-02-20T09:05:35.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:35.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:35.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:35.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:35.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:35.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:35.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:35.430+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:35.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:35.191+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.192655+0000) 2024-02-20T09:05:35.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:35.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:35.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:35.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:35.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:35.453+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:35.454+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:35.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:35.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:35.644+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:35.645+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:35.683+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.684421+0000) 2024-02-20T09:05:36.131 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:36.008+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55088 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:36.132 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:36.006+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.007622+0000) 2024-02-20T09:05:36.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:36.008+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.009288+0000) 2024-02-20T09:05:36.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:36.006+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.007377+0000) 2024-02-20T09:05:36.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:36.191+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.192876+0000) 2024-02-20T09:05:36.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:36.381+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.382311+0000) 2024-02-20T09:05:36.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:36.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:36.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:36.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:36.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:36.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:36.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:36.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:36.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:36.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:36.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:36.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:36.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:36.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:36.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:36.504+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:36.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:36.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:36.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:36.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:36.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:36.683+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.684637+0000) 2024-02-20T09:05:37.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:37.195+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:37.196274+0000) 2024-02-20T09:05:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:37.384+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:37.384934+0000) 2024-02-20T09:05:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:37.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:37.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:37.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:37.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:37.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:37.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:37.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:37.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:37.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:37.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:37.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:37.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:37.617+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:37.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:37.686+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:37.687026+0000) 2024-02-20T09:05:38.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:38.198+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:38.199740+0000) 2024-02-20T09:05:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:38.386+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:38.387069+0000) 2024-02-20T09:05:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:38.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:38.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:38.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:38.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:38.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:38.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:38.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:38.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:38.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:38.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:38.664+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:38.665+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:38.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:38.688+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:38.689617+0000) 2024-02-20T09:05:39.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:39.201+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:39.202588+0000) 2024-02-20T09:05:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:39.388+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:39.389213+0000) 2024-02-20T09:05:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:39.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:39.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:39.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:39.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:39.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:39.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:39.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:39.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:39.654+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:39.690+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:39.691522+0000) 2024-02-20T09:05:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:39.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:39.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:39.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:39.557+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:40.201+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.202765+0000) 2024-02-20T09:05:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:40.388+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.389397+0000) 2024-02-20T09:05:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:40.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:40.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:40.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:40.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:40.423+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:40.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:40.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:40.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:40.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:40.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:40.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:40.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:40.534+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:40.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:40.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:40.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:40.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:40.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:40.669+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:40.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:40.670+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:40.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:40.690+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.691697+0000) 2024-02-20T09:05:41.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:41.012+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.013062+0000) 2024-02-20T09:05:41.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:41.012+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.013287+0000) 2024-02-20T09:05:41.138 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:41.015+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55127 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:41.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:41.013+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.014543+0000) 2024-02-20T09:05:41.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:41.202+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.203040+0000) 2024-02-20T09:05:41.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:41.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:41.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:41.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:41.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:41.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:41.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:41.628+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:41.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:41.388+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.389579+0000) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:41.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:41.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:41.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:41.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:41.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:41.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:41.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:41.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:41.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:41.511+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:41.690+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.691874+0000) 2024-02-20T09:05:42.352 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:42.204+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:42.205443+0000) 2024-02-20T09:05:42.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:42.391+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:42.392219+0000) 2024-02-20T09:05:42.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:42.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:42.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:42.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:42.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:42.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:42.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:42.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:42.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:42.467+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:42.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:42.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:42.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:42.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:42.693+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:42.694378+0000) 2024-02-20T09:05:43.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:43.207+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:43.207966+0000) 2024-02-20T09:05:43.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:43.394+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:43.394974+0000) 2024-02-20T09:05:43.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:43.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:43.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:43.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:43.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:43.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:43.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:43.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:43.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:43.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:43.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:43.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:43.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:43.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:43.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:43.503+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:43.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:43.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:43.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:43.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:43.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:43.560+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:43.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:43.561+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:43.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:43.696+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:43.697026+0000) 2024-02-20T09:05:44.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:44.396+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:44.397849+0000) 2024-02-20T09:05:44.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:44.209+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:44.209955+0000) 2024-02-20T09:05:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:44.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:44.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:44.517+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:44.518+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:44.698+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:44.699044+0000) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:44.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:44.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:44.556+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:44.557+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:44.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:44.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:44.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:44.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:44.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:44.510+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:45.397+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.398038+0000) 2024-02-20T09:05:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:45.209+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.210153+0000) 2024-02-20T09:05:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:45.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:45.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:45.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:45.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:45.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:45.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:45.698+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.699226+0000) 2024-02-20T09:05:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:45.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:45.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:45.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:45.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:45.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:45.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:45.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:45.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:46.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:46.021+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55166 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:46.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:46.019+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.020628+0000) 2024-02-20T09:05:46.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:46.019+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.020472+0000) 2024-02-20T09:05:46.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:46.019+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.020468+0000) 2024-02-20T09:05:46.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:46.209+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.210346+0000) 2024-02-20T09:05:46.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:46.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:46.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:46.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:46.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:46.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:46.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:46.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:46.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:46.397+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.398189+0000) 2024-02-20T09:05:46.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:46.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:46.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:46.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:46.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:46.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:46.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:46.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:46.702 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:46.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:46.702 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:46.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:46.702 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:46.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:46.702 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:46.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:46.698+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.699404+0000) 2024-02-20T09:05:47.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:47.399+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:47.400253+0000) 2024-02-20T09:05:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:47.211+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:47.212316+0000) 2024-02-20T09:05:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:47.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:47.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:47.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:47.517+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:47.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:47.701+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:47.702052+0000) 2024-02-20T09:05:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:47.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:47.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:47.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:47.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:47.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:47.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:47.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:47.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:47.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:47.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:47.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:47.537+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:48.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:48.402+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:48.403860+0000) 2024-02-20T09:05:48.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:48.213+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:48.214762+0000) 2024-02-20T09:05:48.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:48.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:48.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:48.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:48.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:48.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:48.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:48.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:48.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:48.703+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:48.704487+0000) 2024-02-20T09:05:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:48.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:48.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:48.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:48.523+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:48.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:48.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:48.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:48.553+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:49.215+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:49.216819+0000) 2024-02-20T09:05:49.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:49.405+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:49.405955+0000) 2024-02-20T09:05:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:49.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:49.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:49.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:49.455+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:49.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:49.706+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:49.707199+0000) 2024-02-20T09:05:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:49.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:49.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:49.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:49.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:49.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:49.550+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:49.550+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:49.550+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:49.551+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:50.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:50.216+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.216951+0000) 2024-02-20T09:05:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:50.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:50.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:50.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:50.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:50.453+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:50.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:50.405+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.406132+0000) 2024-02-20T09:05:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:50.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:50.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:50.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:50.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:50.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:50.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:50.706+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.707379+0000) 2024-02-20T09:05:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:50.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:50.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:50.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:50.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:50.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:51.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:51.027+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.028190+0000) 2024-02-20T09:05:51.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:51.029+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.030070+0000) 2024-02-20T09:05:51.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:51.028+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55205 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:51.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:51.026+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.027243+0000) 2024-02-20T09:05:51.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:51.216+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.217126+0000) 2024-02-20T09:05:51.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:51.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:51.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:51.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:51.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:51.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:51.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:51.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:51.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:51.405+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.406315+0000) 2024-02-20T09:05:51.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:51.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:51.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:51.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:51.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:51.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:51.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:51.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:51.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:51.706+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.707582+0000) 2024-02-20T09:05:51.887 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:51.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:51.887 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:51.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:51.887 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:51.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:51.887 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:51.599+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:52.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:52.218+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:52.219647+0000) 2024-02-20T09:05:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:52.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:52.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:52.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:52.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:52.408+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:52.409693+0000) 2024-02-20T09:05:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:52.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:52.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:52.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:52.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:52.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:52.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:52.708+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:52.709777+0000) 2024-02-20T09:05:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:52.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:52.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:52.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:52.630+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:53.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:53.220+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:53.221804+0000) 2024-02-20T09:05:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:53.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:53.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:53.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:53.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:53.412+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:53.413270+0000) 2024-02-20T09:05:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:53.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:53.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:53.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:53.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:53.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:53.712+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:53.713303+0000) 2024-02-20T09:05:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:53.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:53.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:53.623+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:53.624+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:54.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:54.222+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:54.223917+0000) 2024-02-20T09:05:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:54.414+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:54.415455+0000) 2024-02-20T09:05:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:54.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:54.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:54.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:54.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:54.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:54.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:54.714+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:54.715265+0000) 2024-02-20T09:05:54.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:54.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:54.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:54.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:54.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:54.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:54.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:54.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:54.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:54.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:54.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:54.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:54.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:54.673+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:54.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:54.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:55.414+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.415645+0000) 2024-02-20T09:05:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:55.223+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.224103+0000) 2024-02-20T09:05:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:55.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:55.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:55.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:55.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:55.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:55.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:55.714+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.715468+0000) 2024-02-20T09:05:55.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:55.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:55.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:55.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:55.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:55.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:55.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:55.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:55.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:55.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:55.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:55.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:55.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:55.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:55.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:55.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:56.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:05:56.036+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55244 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:05:56.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:56.033+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.033999+0000) 2024-02-20T09:05:56.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:56.033+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.034826+0000) 2024-02-20T09:05:56.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:56.033+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.034045+0000) 2024-02-20T09:05:56.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:56.223+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.224254+0000) 2024-02-20T09:05:56.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:56.414+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.415847+0000) 2024-02-20T09:05:56.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:56.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:56.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:56.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:56.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:56.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:56.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:56.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:56.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:56.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:56.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:56.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:56.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:56.415+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:56.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:56.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:56.714+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.715649+0000) 2024-02-20T09:05:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:56.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:56.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:56.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:56.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:57.226+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:57.227021+0000) 2024-02-20T09:05:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:57.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:57.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:57.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:57.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:57.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:57.418+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:57.419320+0000) 2024-02-20T09:05:57.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:57.716+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:57.718144+0000) 2024-02-20T09:05:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:57.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:57.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:57.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:57.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:57.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:57.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:57.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:57.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:58.419+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:58.421556+0000) 2024-02-20T09:05:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:58.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:58.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:58.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:58.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:58.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:58.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:58.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:58.228+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:58.230024+0000) 2024-02-20T09:05:58.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:58.718+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:58.720659+0000) 2024-02-20T09:05:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:58.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:58.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:58.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:58.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:58.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:58.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:58.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:58.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:59.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:59.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:59.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:05:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:59.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:05:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:59.421+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:59.423819+0000) 2024-02-20T09:05:59.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:59.230+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:59.231972+0000) 2024-02-20T09:05:59.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:05:59.720+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:59.722727+0000) 2024-02-20T09:05:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:59.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:59.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:59.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:05:59.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:59.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:59.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:59.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:05:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:05:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:05:59.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:00.230+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.232157+0000) 2024-02-20T09:06:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:00.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:00.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:00.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:00.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:00.422+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.423977+0000) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:00.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:00.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:00.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:00.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:00.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:00.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:00.720+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.722903+0000) 2024-02-20T09:06:00.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:00.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:00.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:00.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:00.663+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:01.318 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:01.044+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55283 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:01.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:01.041+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.042023+0000) 2024-02-20T09:06:01.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:01.041+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.042556+0000) 2024-02-20T09:06:01.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:01.040+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.040965+0000) 2024-02-20T09:06:01.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:01.231+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.232348+0000) 2024-02-20T09:06:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:01.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:01.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:01.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:01.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:01.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:01.423+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.424197+0000) 2024-02-20T09:06:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:01.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:01.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:01.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:01.475+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:01.722+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.723089+0000) 2024-02-20T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:01.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:01.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:01.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:01.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:02.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:02.426+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:02.426930+0000) 2024-02-20T09:06:02.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:02.234+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:02.235661+0000) 2024-02-20T09:06:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:02.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:02.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:02.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:02.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:02.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:02.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:02.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:02.725+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:02.726555+0000) 2024-02-20T09:06:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:02.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:02.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:02.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:02.726+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:02.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:02.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:02.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:02.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:03.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:03.428+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:03.429390+0000) 2024-02-20T09:06:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:03.237+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:03.238129+0000) 2024-02-20T09:06:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:03.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:03.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:03.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:03.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:03.728+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:03.729882+0000) 2024-02-20T09:06:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:03.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:03.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:03.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:03.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:03.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:03.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:03.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:03.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:03.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:04.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:04.430+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:04.431756+0000) 2024-02-20T09:06:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:04.240+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:04.241031+0000) 2024-02-20T09:06:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:04.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:04.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:04.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:04.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:04.395+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:04.731+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:04.732793+0000) 2024-02-20T09:06:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:04.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:04.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:04.560+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:04.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:04.560+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:04.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:04.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:04.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:04.675+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:05.431+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.431933+0000) 2024-02-20T09:06:05.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:05.240+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.241203+0000) 2024-02-20T09:06:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:05.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:05.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:05.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:05.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:05.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:05.732+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.732974+0000) 2024-02-20T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:05.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:05.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:05.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:05.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:05.541+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:05.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:05.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:05.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:05.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:06.319 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:06.051+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55322 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:06.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:06.050+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.051046+0000) 2024-02-20T09:06:06.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:06.049+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.050178+0000) 2024-02-20T09:06:06.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:06.051+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.051995+0000) 2024-02-20T09:06:06.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:06.240+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.241397+0000) 2024-02-20T09:06:06.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:06.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:06.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:06.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:06.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:06.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:06.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:06.341+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:06.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:06.431+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.432113+0000) 2024-02-20T09:06:06.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:06.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:06.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:06.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:06.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:06.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:06.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:06.526+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:06.732+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.733128+0000) 2024-02-20T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:06.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:06.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:06.711+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:06.712+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:07.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:07.242+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:07.243791+0000) 2024-02-20T09:06:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:07.434+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:07.435399+0000) 2024-02-20T09:06:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:07.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:07.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:07.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:07.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:07.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:07.734+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:07.735578+0000) 2024-02-20T09:06:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:07.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:07.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:07.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:07.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:07.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:07.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:07.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:07.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:07.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:07.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:07.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:07.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:07.668+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:08.437+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:08.437955+0000) 2024-02-20T09:06:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:08.246+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:08.247196+0000) 2024-02-20T09:06:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:08.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:08.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:08.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:08.362+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:08.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:08.737+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:08.738843+0000) 2024-02-20T09:06:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:08.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:08.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:08.497+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:08.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:08.498+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:08.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:08.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:08.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:08.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:08.624+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:09.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:09.439+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:09.440052+0000) 2024-02-20T09:06:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:09.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:09.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:09.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:09.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:09.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:09.248+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:09.249275+0000) 2024-02-20T09:06:09.743 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:09.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:09.743 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:09.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:09.743 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:09.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:09.743 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:09.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:09.744 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:09.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:09.744 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:09.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:09.744 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:09.469+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:09.744 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:09.470+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:09.740+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:09.741878+0000) 2024-02-20T09:06:10.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:10.248+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.249468+0000) 2024-02-20T09:06:10.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:10.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:10.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:10.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:10.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:10.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:10.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:10.326+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:10.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:10.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:10.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:10.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:10.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:10.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:10.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:10.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:10.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:10.439+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.440233+0000) 2024-02-20T09:06:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:10.741+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.742020+0000) 2024-02-20T09:06:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:10.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:10.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:10.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:10.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:11.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:11.056+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.057862+0000) 2024-02-20T09:06:11.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:11.055+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.055970+0000) 2024-02-20T09:06:11.320 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:11.058+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55361 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:11.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:11.056+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.056972+0000) 2024-02-20T09:06:11.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:11.248+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.249666+0000) 2024-02-20T09:06:11.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:11.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:11.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:11.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:11.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:11.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:11.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:11.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:11.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:11.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:11.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:11.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:11.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:11.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:11.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:11.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:11.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:11.439+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.440424+0000) 2024-02-20T09:06:11.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:11.741+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.742206+0000) 2024-02-20T09:06:11.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:11.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:11.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:11.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:11.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:11.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:11.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:11.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:12.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:12.251+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:12.251955+0000) 2024-02-20T09:06:12.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:12.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:12.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:12.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:12.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:12.401+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:12.442+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:12.443855+0000) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:12.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:12.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:12.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:12.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:12.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:12.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:12.743+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:12.744559+0000) 2024-02-20T09:06:12.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:12.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:12.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:12.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:12.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:12.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:12.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:12.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:13.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:13.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:13.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:13.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:13.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:13.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:13.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:13.359+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:13.519 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:13.253+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:13.254673+0000) 2024-02-20T09:06:13.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:13.445+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:13.446924+0000) 2024-02-20T09:06:13.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:13.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:13.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:13.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:13.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:13.746+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:13.747156+0000) 2024-02-20T09:06:13.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:13.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:13.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:13.516+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:13.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:13.596+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:13.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:13.596+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:13.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:13.596+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:13.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:13.597+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:14.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:14.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:14.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:14.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:14.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:14.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:14.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:14.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:14.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:14.448+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:14.449007+0000) 2024-02-20T09:06:14.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:14.255+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:14.256497+0000) 2024-02-20T09:06:14.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:14.749+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:14.750139+0000) 2024-02-20T09:06:14.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:14.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:14.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:14.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:14.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:14.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:14.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:14.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:14.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:14.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:14.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:14.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:15.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:15.448+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.449178+0000) 2024-02-20T09:06:15.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:15.255+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.256682+0000) 2024-02-20T09:06:15.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:15.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:15.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:15.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:15.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:15.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:15.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:15.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:15.871 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:15.749+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.750319+0000) 2024-02-20T09:06:15.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:15.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:15.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:15.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:15.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:15.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:15.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:15.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:15.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:15.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:15.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:15.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:15.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:15.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:15.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:15.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:16.185 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:16.065+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55400 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:16.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:16.063+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.064916+0000) 2024-02-20T09:06:16.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:16.063+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.064381+0000) 2024-02-20T09:06:16.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:16.062+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.063369+0000) 2024-02-20T09:06:16.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:16.255+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.256871+0000) 2024-02-20T09:06:16.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:16.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:16.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:16.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:16.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:16.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:16.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:16.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:16.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:16.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:16.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:16.534+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:16.448+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.449381+0000) 2024-02-20T09:06:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:16.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:16.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:16.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:16.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:16.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:16.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:16.749+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.750515+0000) 2024-02-20T09:06:17.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:17.450+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:17.451663+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:17.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:17.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:17.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:17.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:17.258+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:17.259810+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:17.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:17.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:17.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:17.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:17.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:17.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:17.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:17.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:17.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:17.544+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:17.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:17.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:17.754 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:17.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:17.751+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:17.752557+0000) 2024-02-20T09:06:18.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:18.452+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:18.453758+0000) 2024-02-20T09:06:18.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:18.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:18.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:18.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:18.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:18.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:18.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:18.457+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:18.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:18.261+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:18.262365+0000) 2024-02-20T09:06:18.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:18.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:18.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:18.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:18.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:18.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:18.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:18.479+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:18.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:18.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:18.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:18.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:18.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:18.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:18.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:18.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:18.754+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:18.755275+0000) 2024-02-20T09:06:19.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:19.454+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:19.455752+0000) 2024-02-20T09:06:19.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:19.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:19.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:19.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:19.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:19.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:19.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:19.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:19.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:19.263+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:19.264528+0000) 2024-02-20T09:06:19.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:19.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:19.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:19.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:19.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:19.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:19.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:19.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:19.756+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:19.757245+0000) 2024-02-20T09:06:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:19.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:19.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:19.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:19.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:20.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:20.455+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.455949+0000) 2024-02-20T09:06:20.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:20.263+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.264727+0000) 2024-02-20T09:06:20.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:20.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:20.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:20.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:20.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:20.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:20.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:20.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:20.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:20.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:20.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:20.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:20.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:20.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:20.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:20.497+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:20.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:20.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:20.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:20.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:20.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:20.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:20.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:20.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:21.068 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:20.756+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.757422+0000) 2024-02-20T09:06:21.320 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:21.072+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55439 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:21.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:21.069+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.070341+0000) 2024-02-20T09:06:21.321 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:21.070+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.071502+0000) 2024-02-20T09:06:21.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:21.071+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.072675+0000) 2024-02-20T09:06:21.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:21.263+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.264921+0000) 2024-02-20T09:06:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:21.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:21.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:21.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:21.405+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:21.455+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.456121+0000) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:21.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:21.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:21.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:21.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:21.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:21.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:21.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:21.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:21.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:21.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:22.015 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:21.756+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.757636+0000) 2024-02-20T09:06:22.535 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:22.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:22.535 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:22.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:22.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:22.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:22.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:22.455+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:22.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:22.457+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:22.458570+0000) 2024-02-20T09:06:22.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:22.266+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:22.266977+0000) 2024-02-20T09:06:22.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:22.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:22.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:22.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:22.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:22.759+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:22.760094+0000) 2024-02-20T09:06:22.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:22.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:22.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:22.534+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:22.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:22.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:22.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:22.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:22.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:22.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:22.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:22.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:23.535 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:23.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:23.535 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:23.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:23.535 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:23.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:23.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:23.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:23.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:23.460+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:23.461609+0000) 2024-02-20T09:06:23.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:23.267+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:23.268752+0000) 2024-02-20T09:06:23.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:23.761+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:23.762675+0000) 2024-02-20T09:06:23.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:23.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:23.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:23.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:23.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:23.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:23.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:23.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:23.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:23.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:23.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:23.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:23.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:23.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:23.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:23.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:24.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:24.269+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:24.270864+0000) 2024-02-20T09:06:24.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:24.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:24.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:24.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:24.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:24.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:24.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:24.452+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:24.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:24.463+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:24.464853+0000) 2024-02-20T09:06:24.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:24.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:24.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:24.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:24.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:24.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:24.766 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:24.500+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:24.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:24.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:24.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:24.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:24.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:24.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:24.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:24.588+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:25.087 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:24.763+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:24.764768+0000) 2024-02-20T09:06:25.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:25.270+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.271046+0000) 2024-02-20T09:06:25.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:25.464+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.465049+0000) 2024-02-20T09:06:25.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:25.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:25.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:25.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:25.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:25.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:25.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:25.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:25.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:25.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:25.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:25.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:25.453+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:25.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:25.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:25.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:25.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:25.604+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:25.764+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.764958+0000) 2024-02-20T09:06:26.356 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:26.079+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55478 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:26.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:26.077+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.078538+0000) 2024-02-20T09:06:26.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:26.077+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.078195+0000) 2024-02-20T09:06:26.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:26.078+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.079282+0000) 2024-02-20T09:06:26.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:26.270+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.271219+0000) 2024-02-20T09:06:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:26.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:26.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:26.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:26.583+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:26.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:26.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:26.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:26.494+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:26.464+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.465200+0000) 2024-02-20T09:06:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:26.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:26.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:26.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:26.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:26.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:26.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:26.764+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.765114+0000) 2024-02-20T09:06:27.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:27.467+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:27.467975+0000) 2024-02-20T09:06:27.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:27.272+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:27.273731+0000) 2024-02-20T09:06:27.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:27.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:27.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:27.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:27.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:27.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:27.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:27.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:27.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:27.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:27.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:27.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:27.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:27.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:27.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:27.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:27.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:27.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:27.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:27.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:27.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:27.553+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:27.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:27.554+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:27.767+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:27.768655+0000) 2024-02-20T09:06:28.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:28.275+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:28.276082+0000) 2024-02-20T09:06:28.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:28.469+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:28.470791+0000) 2024-02-20T09:06:28.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:28.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:28.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:28.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:28.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:28.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:28.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:28.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:28.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:28.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:28.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:28.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:28.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:28.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:28.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:28.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:28.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:28.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:28.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:28.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:28.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:28.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:28.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:28.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:28.771+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:28.772135+0000) 2024-02-20T09:06:29.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:29.277+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:29.278331+0000) 2024-02-20T09:06:29.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:29.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:29.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:29.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:29.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:29.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:29.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:29.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:29.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:29.472+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:29.473862+0000) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:29.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:29.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:29.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:29.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:29.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:29.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:29.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:29.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:29.778 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:29.604+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:29.773+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:29.774584+0000) 2024-02-20T09:06:30.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:30.277+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.278520+0000) 2024-02-20T09:06:30.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:30.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:30.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:30.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:30.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:30.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:30.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:30.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:30.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:30.473+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.474029+0000) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:30.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:30.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:30.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:30.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:30.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:30.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:30.777 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:30.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:30.778 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:30.605+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:31.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:30.773+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.774854+0000) 2024-02-20T09:06:31.357 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:31.086+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55517 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:31.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:31.085+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.086455+0000) 2024-02-20T09:06:31.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:31.083+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.084585+0000) 2024-02-20T09:06:31.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:31.084+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.085491+0000) 2024-02-20T09:06:31.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:31.277+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.278710+0000) 2024-02-20T09:06:31.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:31.473+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.474227+0000) 2024-02-20T09:06:31.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:31.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:31.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:31.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:31.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:31.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:31.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:31.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:31.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:31.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:31.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:31.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:31.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:31.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:31.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:31.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:31.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:31.774+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.775063+0000) 2024-02-20T09:06:31.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:31.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:31.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:31.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:31.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:31.612+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:31.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:31.613+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:32.478 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:32.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:32.478 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:32.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:32.478 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:32.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:32.478 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:32.460+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:32.479 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:32.280+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:32.281116+0000) 2024-02-20T09:06:32.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:32.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:32.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:32.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:32.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:32.583+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:32.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:32.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:32.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:32.475+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:32.476728+0000) 2024-02-20T09:06:32.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:32.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:32.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:32.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:32.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:32.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:32.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:32.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:32.776+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:32.777799+0000) 2024-02-20T09:06:33.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:33.282+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:33.283606+0000) 2024-02-20T09:06:33.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:33.478+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:33.479342+0000) 2024-02-20T09:06:33.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:33.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:33.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:33.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:33.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:33.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:33.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:33.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:33.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:33.596+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:33.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:33.596+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:33.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:33.596+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:33.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:33.597+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:33.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:33.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:33.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:33.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:33.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:33.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:33.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:33.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:33.779+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:33.780445+0000) 2024-02-20T09:06:34.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:34.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:34.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:34.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:34.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:34.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:34.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:34.467+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:34.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:34.284+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:34.285895+0000) 2024-02-20T09:06:34.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:34.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:34.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:34.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:34.585+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:34.480+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:34.481422+0000) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:34.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:34.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:34.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:34.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:34.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:34.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:34.782+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:34.782960+0000) 2024-02-20T09:06:35.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:35.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:35.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:35.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:35.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:35.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:35.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:35.470+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:35.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:35.285+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.286115+0000) 2024-02-20T09:06:35.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:35.474+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:35.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:35.474+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:35.784 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:35.474+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:35.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:35.475+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:35.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:35.480+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.481604+0000) 2024-02-20T09:06:35.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:35.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:35.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:35.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:35.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:35.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:35.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:35.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:36.089 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:35.782+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.783231+0000) 2024-02-20T09:06:36.357 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:36.093+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55556 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:36.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:36.091+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.092317+0000) 2024-02-20T09:06:36.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:36.091+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.092641+0000) 2024-02-20T09:06:36.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:36.092+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.093469+0000) 2024-02-20T09:06:36.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:36.285+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.286299+0000) 2024-02-20T09:06:36.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:36.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:36.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:36.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:36.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:36.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:36.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:36.444+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:36.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:36.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:36.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:36.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:36.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:36.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:36.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:36.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:36.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:36.480+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.481914+0000) 2024-02-20T09:06:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:36.782+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.783417+0000) 2024-02-20T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:36.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:36.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:36.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:36.647+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:37.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:37.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:37.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:37.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:37.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:37.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:37.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:37.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:37.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:37.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:37.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:37.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:37.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:37.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:37.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:37.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:37.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:37.483+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:37.484055+0000) 2024-02-20T09:06:37.487 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:37.287+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:37.288696+0000) 2024-02-20T09:06:37.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:37.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:37.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:37.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:37.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:37.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:37.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:37.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:37.785+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:37.786022+0000) 2024-02-20T09:06:38.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:38.291+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:38.292318+0000) 2024-02-20T09:06:38.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:38.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:38.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:38.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:38.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:38.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:38.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:38.507+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:38.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:38.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:38.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:38.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:38.486+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:38.487408+0000) 2024-02-20T09:06:38.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:38.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:38.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:38.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:38.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:38.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:38.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:38.655+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:38.788+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:38.789344+0000) 2024-02-20T09:06:39.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:39.293+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:39.294526+0000) 2024-02-20T09:06:39.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:39.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:39.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:39.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:39.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:39.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:39.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:39.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:39.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:39.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:39.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:39.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:39.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:39.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:39.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:39.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:39.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:39.489+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:39.490324+0000) 2024-02-20T09:06:39.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:39.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:39.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:39.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:39.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:39.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:39.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:39.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:40.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:39.792+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:39.792969+0000) 2024-02-20T09:06:40.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:40.293+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.294715+0000) 2024-02-20T09:06:40.492 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:40.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:40.492 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:40.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:40.492 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:40.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:40.492 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:40.470+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:40.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:40.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:40.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:40.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:40.493 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:40.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:40.493 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:40.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:40.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:40.489+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.490509+0000) 2024-02-20T09:06:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:40.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:40.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:40.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:40.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:41.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:40.792+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.793151+0000) 2024-02-20T09:06:41.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:41.100+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55595 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:41.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:41.098+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.099373+0000) 2024-02-20T09:06:41.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:41.098+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.099797+0000) 2024-02-20T09:06:41.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:41.097+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.098852+0000) 2024-02-20T09:06:41.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:41.294+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.294967+0000) 2024-02-20T09:06:41.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:41.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:41.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:41.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:41.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:41.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:41.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:41.493+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:41.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:41.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:41.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:41.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:41.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:41.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:41.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:41.382+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:41.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:41.489+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.490729+0000) 2024-02-20T09:06:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:41.792+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.793336+0000) 2024-02-20T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:41.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:41.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:41.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:41.690+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:42.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:42.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:42.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:42.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:42.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:42.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:42.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:42.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:42.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:42.296+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:42.297473+0000) 2024-02-20T09:06:42.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:42.492+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:42.493182+0000) 2024-02-20T09:06:42.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:42.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:42.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:42.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:42.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:42.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:42.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:42.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:42.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:42.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:42.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:42.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:42.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:42.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:42.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:42.661+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:42.795+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:42.796865+0000) 2024-02-20T09:06:43.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:43.298+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:43.299908+0000) 2024-02-20T09:06:43.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:43.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:43.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:43.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:43.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:43.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:43.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:43.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:43.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:43.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:43.520+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:43.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:43.521+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:43.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:43.495+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:43.496665+0000) 2024-02-20T09:06:43.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:43.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:43.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:43.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:43.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:43.710+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:43.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:43.711+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:44.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:43.798+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:43.799494+0000) 2024-02-20T09:06:44.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:44.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:44.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:44.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:44.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:44.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:44.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:44.348+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:44.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:44.301+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:44.302060+0000) 2024-02-20T09:06:44.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:44.497+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:44.498864+0000) 2024-02-20T09:06:44.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:44.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:44.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:44.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:44.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:44.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:44.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:44.536+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:44.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:44.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:44.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:44.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:44.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:44.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:44.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:44.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:44.800+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:44.801675+0000) 2024-02-20T09:06:45.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:45.301+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.302255+0000) 2024-02-20T09:06:45.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:45.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:45.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:45.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:45.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:45.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:45.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:45.391+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:45.498+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.499131+0000) 2024-02-20T09:06:45.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:45.557+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:45.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:45.557+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:45.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:45.557+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:45.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:45.558+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:46.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:45.800+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.801914+0000) 2024-02-20T09:06:46.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:45.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:46.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:45.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:46.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:45.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:46.105 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:45.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:46.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:46.107+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55634 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:46.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:46.105+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.106857+0000) 2024-02-20T09:06:46.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:46.105+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.106235+0000) 2024-02-20T09:06:46.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:46.106+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.107845+0000) 2024-02-20T09:06:46.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:46.301+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.302406+0000) 2024-02-20T09:06:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:46.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:46.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:46.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:46.533+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:46.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:46.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:46.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:46.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:46.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:46.498+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.499307+0000) 2024-02-20T09:06:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:46.801+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.802192+0000) 2024-02-20T09:06:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:46.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:46.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:46.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:46.777+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:47.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:47.304+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:47.305112+0000) 2024-02-20T09:06:47.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:47.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:47.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:47.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:47.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:47.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:47.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:47.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:47.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:47.500+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:47.501813+0000) 2024-02-20T09:06:47.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:47.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:47.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:47.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:47.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:47.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:47.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:47.562+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:47.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:47.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:48.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:47.803+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:47.804896+0000) 2024-02-20T09:06:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:47.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:47.769+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:47.770+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:48.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:48.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:48.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:48.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:48.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:48.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:48.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:48.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:48.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:48.306+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:48.307558+0000) 2024-02-20T09:06:48.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:48.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:48.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:48.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:48.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:48.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:48.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:48.580+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:48.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:48.503+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:48.504515+0000) 2024-02-20T09:06:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:48.806+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:48.807363+0000) 2024-02-20T09:06:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:48.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:48.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:48.761+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:48.762+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:49.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:49.308+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:49.309733+0000) 2024-02-20T09:06:49.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:49.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:49.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:49.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:49.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:49.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:49.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:49.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:49.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:49.506+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:49.507609+0000) 2024-02-20T09:06:49.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:49.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:49.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:49.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:49.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:49.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:49.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:49.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:49.808+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:49.809655+0000) 2024-02-20T09:06:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:49.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:49.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:49.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:49.785+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:50.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:50.308+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.309918+0000) 2024-02-20T09:06:50.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:50.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:50.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:50.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:50.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:50.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:50.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:50.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:50.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:50.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:50.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:50.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:50.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:50.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:50.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:50.657+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:50.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:50.505+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.507783+0000) 2024-02-20T09:06:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:50.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:50.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:50.786+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:51.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:50.807+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.809845+0000) 2024-02-20T09:06:51.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:50.787+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:51.420 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:51.113+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55673 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:51.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:51.112+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.114310+0000) 2024-02-20T09:06:51.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:51.110+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.112485+0000) 2024-02-20T09:06:51.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:51.308+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.310109+0000) 2024-02-20T09:06:51.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:51.112+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.114739+0000) 2024-02-20T09:06:51.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:51.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:51.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:51.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:51.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:51.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:51.663+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:51.663+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:51.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:51.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:51.506+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.507949+0000) 2024-02-20T09:06:52.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:51.808+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.809975+0000) 2024-02-20T09:06:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:51.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:51.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:51.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:51.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:52.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:52.310+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:52.312787+0000) 2024-02-20T09:06:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:52.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:52.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:52.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:52.398+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:52.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:52.509+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:52.510955+0000) 2024-02-20T09:06:52.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:52.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:52.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:52.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:52.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:52.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:52.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:52.712+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:52.811+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:52.812955+0000) 2024-02-20T09:06:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:52.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:52.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:52.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:52.831+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:53.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:53.313+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:53.315203+0000) 2024-02-20T09:06:53.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:53.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:53.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:53.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:53.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:53.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:53.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:53.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:53.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:53.511+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:53.513514+0000) 2024-02-20T09:06:53.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:53.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:53.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:53.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:53.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:53.680+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:53.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:53.681+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:53.816+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:53.817032+0000) 2024-02-20T09:06:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:53.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:53.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:53.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:53.805+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:54.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:54.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:54.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:54.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:54.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:54.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:54.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:54.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:54.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:54.317+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:54.318029+0000) 2024-02-20T09:06:54.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:54.515+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:54.516182+0000) 2024-02-20T09:06:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:54.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:54.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:54.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:54.714+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:54.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:54.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:54.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:54.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:54.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:54.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:54.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:54.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:54.818+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:54.819439+0000) 2024-02-20T09:06:55.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:55.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:55.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:55.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:55.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:55.428+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:55.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:55.429+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:55.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:55.317+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.318318+0000) 2024-02-20T09:06:55.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:55.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:55.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:55.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:55.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:55.757+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:55.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:55.758+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:55.515+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.516372+0000) 2024-02-20T09:06:56.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:55.818+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.819686+0000) 2024-02-20T09:06:56.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:55.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:56.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:55.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:56.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:55.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:56.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:55.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:56.407 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:06:56.122+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55712 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:06:56.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:56.119+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.120140+0000) 2024-02-20T09:06:56.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:56.118+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.119843+0000) 2024-02-20T09:06:56.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:56.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:56.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:56.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:56.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:56.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:56.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:56.120+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.121397+0000) 2024-02-20T09:06:56.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:56.317+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.318498+0000) 2024-02-20T09:06:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:56.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:56.515+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.516565+0000) 2024-02-20T09:06:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:56.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:56.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:56.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:57.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:56.788+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:56.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:56.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:56.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:56.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:57.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:56.818+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.819862+0000) 2024-02-20T09:06:57.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:57.320+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:57.321886+0000) 2024-02-20T09:06:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:57.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:57.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:57.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:57.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:57.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:57.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:57.519+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:57.520117+0000) 2024-02-20T09:06:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:57.816+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:57.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:57.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:57.817+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:57.780+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:57.780+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:57.780+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:57.781+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:58.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:57.822+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:57.823400+0000) 2024-02-20T09:06:58.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:58.324+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:58.324951+0000) 2024-02-20T09:06:58.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:58.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:58.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:58.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:58.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:58.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:58.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:58.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:58.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:58.521+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:58.522706+0000) 2024-02-20T09:06:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:58.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:58.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:58.825+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:06:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:58.826+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:58.826934+0000) 2024-02-20T09:06:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:58.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:58.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:58.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:58.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:06:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:58.804+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:06:59.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:59.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:59.327288+0000) 2024-02-20T09:06:59.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:59.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:59.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:59.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:59.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:59.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:06:59.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:59.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:59.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:06:59.524+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:59.525767+0000) 2024-02-20T09:07:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:59.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:59.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:59.785+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:06:59.786+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:59.828+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:59.828963+0000) 2024-02-20T09:07:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:59.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:59.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:59.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:00.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:06:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:06:59.834+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:00.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:00.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.327483+0000) 2024-02-20T09:07:00.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:00.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:00.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:00.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:00.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:00.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:00.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:00.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:00.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:00.525+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.525974+0000) 2024-02-20T09:07:00.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:00.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:00.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:00.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:00.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:00.768+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:00.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:00.769+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:01.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:00.828+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.829129+0000) 2024-02-20T09:07:01.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:00.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:01.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:00.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:01.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:00.877+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:01.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:00.878+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:01.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:01.126+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.127728+0000) 2024-02-20T09:07:01.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:01.127+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.127976+0000) 2024-02-20T09:07:01.446 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:01.128+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55751 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:01.125+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.126604+0000) 2024-02-20T09:07:01.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:01.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.327654+0000) 2024-02-20T09:07:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:01.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:01.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:01.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:01.444+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:01.525+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.526170+0000) 2024-02-20T09:07:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:01.828+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.829291+0000) 2024-02-20T09:07:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:01.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:01.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:01.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:01.854+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:01.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:01.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:01.758+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:01.759+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:02.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:02.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:02.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:02.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:02.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:02.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:02.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:02.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:02.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:02.329+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:02.330060+0000) 2024-02-20T09:07:02.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:02.527+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:02.528804+0000) 2024-02-20T09:07:02.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:02.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:02.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:02.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:02.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:02.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:03.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:02.784+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:02.831+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:02.832657+0000) 2024-02-20T09:07:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:02.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:02.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:02.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:02.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:03.532 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:03.331+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:03.332692+0000) 2024-02-20T09:07:03.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:03.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:03.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:03.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:03.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:03.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:03.533 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:03.444+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:03.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:03.530+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:03.531391+0000) 2024-02-20T09:07:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:03.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:03.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:03.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:03.790+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:04.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:03.834+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:03.834948+0000) 2024-02-20T09:07:04.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:03.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:04.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:03.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:03.852+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:03.852+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:04.535 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:04.334+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:04.335651+0000) 2024-02-20T09:07:04.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:04.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:04.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:04.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:04.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:04.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:04.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:04.462+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:04.799 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:04.532+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:04.533860+0000) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:04.836+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:04.837337+0000) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:04.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:04.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:04.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:04.854+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:04.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:04.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:04.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:04.819+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:05.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:05.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:05.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:05.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:05.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:05.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:05.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:05.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:05.536 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:05.334+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.335829+0000) 2024-02-20T09:07:05.799 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:05.533+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.534017+0000) 2024-02-20T09:07:06.132 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:05.836+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.837522+0000) 2024-02-20T09:07:06.132 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:05.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:05.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:05.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:05.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:05.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:05.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:05.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:06.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:05.854+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:06.424 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:06.136+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55790 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:06.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:06.133+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.134617+0000) 2024-02-20T09:07:06.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:06.134+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.135062+0000) 2024-02-20T09:07:06.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:06.335+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.335977+0000) 2024-02-20T09:07:06.425 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:06.133+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.134445+0000) 2024-02-20T09:07:06.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:06.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:06.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:06.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:06.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:06.422+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:06.423+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:06.533+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.534199+0000) 2024-02-20T09:07:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:06.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:06.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:06.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:06.884+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:06.836+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.837714+0000) 2024-02-20T09:07:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:06.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:06.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:06.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:07.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:06.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:07.337+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:07.338631+0000) 2024-02-20T09:07:07.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:07.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:07.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:07.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:07.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:07.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:07.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:07.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:07.799 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:07.535+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:07.536403+0000) 2024-02-20T09:07:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:07.839+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:07.840267+0000) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:07.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:07.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:07.882+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:07.883+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:07.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:07.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:07.836+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:07.837+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:08.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:08.340+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:08.341304+0000) 2024-02-20T09:07:08.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:08.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:08.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:08.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:08.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:08.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:08.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:08.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:08.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:08.537+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:08.538775+0000) 2024-02-20T09:07:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:08.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:08.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:08.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:08.836+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:08.841+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:08.842875+0000) 2024-02-20T09:07:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:08.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:08.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:08.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:09.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:08.901+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:09.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:09.342+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:09.343521+0000) 2024-02-20T09:07:09.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:09.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:09.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:09.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:09.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:09.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:09.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:09.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:09.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:09.539+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:09.540832+0000) 2024-02-20T09:07:10.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:09.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:09.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:09.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:09.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:09.844+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:09.845913+0000) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:09.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:09.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:09.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:10.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:09.935+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:10.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:10.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:10.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:10.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:10.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:10.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:10.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:10.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:10.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:10.342+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.343685+0000) 2024-02-20T09:07:10.799 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:10.540+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.541120+0000) 2024-02-20T09:07:11.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:10.845+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.846158+0000) 2024-02-20T09:07:11.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:10.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:11.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:10.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:11.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:10.975+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:11.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:10.977+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:11.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:10.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:11.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:10.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:11.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:10.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:11.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:10.873+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:11.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:11.141+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.141996+0000) 2024-02-20T09:07:11.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:11.141+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.142426+0000) 2024-02-20T09:07:11.433 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:11.143+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55829 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:11.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:11.141+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.142106+0000) 2024-02-20T09:07:11.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:11.342+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.343879+0000) 2024-02-20T09:07:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:11.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:11.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:11.430+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:11.431+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:11.540+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.541315+0000) 2024-02-20T09:07:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:11.845+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.846311+0000) 2024-02-20T09:07:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:11.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:11.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:11.965+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:11.965+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:11.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:11.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:11.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:11.877+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:12.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:12.346+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:12.347560+0000) 2024-02-20T09:07:12.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:12.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:12.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:12.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:12.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:12.448+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:12.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:12.449+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:12.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:12.543+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:12.543968+0000) 2024-02-20T09:07:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:12.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:12.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:12.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:12.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:12.848+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:12.849727+0000) 2024-02-20T09:07:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:12.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:12.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:12.988+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:13.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:12.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:13.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:13.350+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:13.351167+0000) 2024-02-20T09:07:13.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:13.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:13.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:13.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:13.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:13.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:13.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:13.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:13.800 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:13.546+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:13.546963+0000) 2024-02-20T09:07:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:13.851+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:13.852253+0000) 2024-02-20T09:07:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:13.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:13.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:13.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:13.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:13.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:13.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:13.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:13.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:14.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:14.352+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:14.353437+0000) 2024-02-20T09:07:14.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:14.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:14.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:14.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:14.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:14.417+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:14.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:14.418+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:14.856 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:14.548+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:14.549102+0000) 2024-02-20T09:07:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:14.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:14.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:14.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:15.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:14.935+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:14.853+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:14.854852+0000) 2024-02-20T09:07:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:14.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:15.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:14.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:15.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:14.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:15.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:14.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:15.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:15.352+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.353622+0000) 2024-02-20T09:07:15.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:15.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:15.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:15.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:15.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:15.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:15.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:15.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:15.856 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:15.548+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.549287+0000) 2024-02-20T09:07:16.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:15.854+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.855024+0000) 2024-02-20T09:07:16.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:15.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:16.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:15.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:16.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:15.897+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:16.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:15.898+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:16.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:15.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:16.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:15.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:16.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:15.951+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:16.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:15.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:16.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:16.151+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55868 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:16.148+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.149072+0000) 2024-02-20T09:07:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:16.148+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.149325+0000) 2024-02-20T09:07:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:16.352+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.353807+0000) 2024-02-20T09:07:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:16.147+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.148850+0000) 2024-02-20T09:07:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:16.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:16.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:16.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:16.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:16.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:16.856 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:16.548+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.549476+0000) 2024-02-20T09:07:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:16.854+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.855262+0000) 2024-02-20T09:07:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:16.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:16.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:16.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:16.905+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:16.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:16.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:17.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:16.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:17.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:16.959+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:17.355+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:17.356256+0000) 2024-02-20T09:07:17.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:17.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:17.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:17.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:17.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:17.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:17.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:17.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:17.800 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:17.551+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:17.552128+0000) 2024-02-20T09:07:18.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:17.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:17.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:17.961+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:17.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:17.856+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:17.857913+0000) 2024-02-20T09:07:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:17.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:17.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:17.933+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:18.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:17.934+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:18.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:18.357+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:18.358697+0000) 2024-02-20T09:07:18.555 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:18.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:18.555 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:18.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:18.555 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:18.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:18.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:18.342+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:18.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:18.553+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:18.554135+0000) 2024-02-20T09:07:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:18.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:18.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:18.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:18.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:18.860+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:18.861476+0000) 2024-02-20T09:07:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:18.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:18.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:18.926+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:19.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:18.927+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:19.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:19.360+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:19.361670+0000) 2024-02-20T09:07:19.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:19.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:19.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:19.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:19.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:19.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:19.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:19.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:19.864 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:19.556+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:19.557210+0000) 2024-02-20T09:07:20.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:20.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:20.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:20.028+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:20.029+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:19.862+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:19.863531+0000) 2024-02-20T09:07:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:19.885+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:19.885+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:19.885+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:20.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:19.885+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:20.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:20.360+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.361853+0000) 2024-02-20T09:07:20.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:20.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:20.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:20.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:20.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:20.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:20.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:20.336+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:20.864 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:20.556+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.557376+0000) 2024-02-20T09:07:21.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:21.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:21.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:21.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:21.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:20.862+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.863736+0000) 2024-02-20T09:07:21.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:20.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:21.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:20.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:21.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:20.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:21.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:20.915+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:21.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:21.156+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.157584+0000) 2024-02-20T09:07:21.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:21.158+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55907 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.159+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.159964+0000) 2024-02-20T09:07:21.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.361+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.362040+0000) 2024-02-20T09:07:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:21.156+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.157926+0000) 2024-02-20T09:07:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:21.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:21.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:21.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:21.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:21.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:21.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:21.556+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.557555+0000) 2024-02-20T09:07:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.978+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:21.979+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:21.863+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.864040+0000) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:21.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:21.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:21.908+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:22.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:21.909+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:22.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:22.364+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:22.365396+0000) 2024-02-20T09:07:22.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:22.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:22.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:22.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:22.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:22.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:22.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:22.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:22.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:22.558+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:22.559686+0000) 2024-02-20T09:07:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:23.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:23.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:23.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:23.001+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:23.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:22.865+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:22.866517+0000) 2024-02-20T09:07:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:22.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:22.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:22.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:23.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:22.957+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:23.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:23.367+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:23.367988+0000) 2024-02-20T09:07:23.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:23.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:23.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:23.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:23.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:23.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:23.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:23.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:23.870 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:23.561+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:23.562336+0000) 2024-02-20T09:07:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:24.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:24.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:24.044+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:24.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:24.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:23.868+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:23.869009+0000) 2024-02-20T09:07:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:23.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:23.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:23.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:24.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:23.990+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:24.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:24.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:24.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:24.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:24.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:24.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:24.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:24.439+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:24.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:24.369+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:24.370226+0000) 2024-02-20T09:07:24.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:24.563+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:24.564820+0000) 2024-02-20T09:07:25.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:24.870+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:24.871190+0000) 2024-02-20T09:07:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:25.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:25.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:25.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:25.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:25.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:25.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:25.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:25.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:25.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:25.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:25.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:25.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:25.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:25.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:25.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:25.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:25.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:25.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:25.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:25.369+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.370409+0000) 2024-02-20T09:07:25.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:25.564+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.565022+0000) 2024-02-20T09:07:26.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:25.870+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.871371+0000) 2024-02-20T09:07:26.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:26.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:26.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:26.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:26.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:26.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:26.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:26.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:26.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:26.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:26.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:26.166+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55946 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:26.163+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.164219+0000) 2024-02-20T09:07:26.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:26.165+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.166107+0000) 2024-02-20T09:07:26.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:26.369+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.370597+0000) 2024-02-20T09:07:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:26.162+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.163581+0000) 2024-02-20T09:07:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:26.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:26.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:26.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:26.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:26.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:26.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:26.801 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:26.564+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.565251+0000) 2024-02-20T09:07:27.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:26.870+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.871562+0000) 2024-02-20T09:07:27.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:27.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:27.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:27.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:27.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:27.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:27.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:27.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:27.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.373+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:27.374197+0000) 2024-02-20T09:07:27.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:27.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:27.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:27.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:27.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:27.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:27.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:27.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:27.567+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:27.568509+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:27.874+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:27.875095+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:28.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:28.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:28.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:28.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:28.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:27.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:28.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:28.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:28.375+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:28.376644+0000) 2024-02-20T09:07:28.801 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:28.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:28.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:28.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:28.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:28.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:28.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:28.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:28.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:28.570+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:28.570970+0000) 2024-02-20T09:07:29.200 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:28.876+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:28.877082+0000) 2024-02-20T09:07:29.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:28.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:29.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:28.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:29.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:28.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:29.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:28.959+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:29.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:29.377+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:29.378492+0000) 2024-02-20T09:07:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:29.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:29.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:29.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:29.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:29.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:29.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:29.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:29.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:29.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:29.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:29.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:29.496+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:29.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:29.573+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:29.574199+0000) 2024-02-20T09:07:30.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:29.879+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:29.880296+0000) 2024-02-20T09:07:30.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:30.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:30.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:30.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:30.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:30.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:30.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:30.004+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:30.377+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.378670+0000) 2024-02-20T09:07:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:30.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:30.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:30.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:30.168+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:30.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:30.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:30.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:30.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:30.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:30.545+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:30.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:30.546+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:30.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:30.573+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.574375+0000) 2024-02-20T09:07:31.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:30.879+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.880487+0000) 2024-02-20T09:07:31.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:31.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:31.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:31.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:31.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:31.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:31.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:31.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:31.172+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.173414+0000) 2024-02-20T09:07:31.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:31.173+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 55985 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:31.169+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.170537+0000) 2024-02-20T09:07:31.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:31.377+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.378868+0000) 2024-02-20T09:07:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:31.170+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.171246+0000) 2024-02-20T09:07:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:31.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:31.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:31.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:31.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:31.178+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:31.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:31.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:31.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:31.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:31.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:31.546+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:31.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:31.547+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:31.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:31.573+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.574555+0000) 2024-02-20T09:07:32.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:31.879+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.880666+0000) 2024-02-20T09:07:32.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:32.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:32.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:32.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:32.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:32.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:32.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:32.226+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:32.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:32.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:32.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:32.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:32.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:32.090+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:32.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:32.091+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:32.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:32.381+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:32.382486+0000) 2024-02-20T09:07:32.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:32.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:32.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:32.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:32.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:32.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:32.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:32.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:32.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:32.576+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:32.577307+0000) 2024-02-20T09:07:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:32.881+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:32.882841+0000) 2024-02-20T09:07:33.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:33.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:33.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:33.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:33.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:33.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:33.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:33.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:33.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:33.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:33.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:33.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:33.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:33.237+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:33.331 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:33.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:33.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:33.383+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:33.384918+0000) 2024-02-20T09:07:33.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:33.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:33.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:33.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:33.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:33.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:33.582 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:33.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:33.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:33.579+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:33.579955+0000) 2024-02-20T09:07:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:33.885+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:33.886161+0000) 2024-02-20T09:07:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:34.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:34.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:34.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:34.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:34.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:34.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:34.387264+0000) 2024-02-20T09:07:34.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:34.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:34.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:34.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:34.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:34.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:34.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:34.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:34.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:34.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:34.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:34.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:34.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:34.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:34.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:34.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:34.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:34.581+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:34.582319+0000) 2024-02-20T09:07:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:34.888+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:34.889213+0000) 2024-02-20T09:07:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.023+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:35.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.387445+0000) 2024-02-20T09:07:35.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:35.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:35.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:35.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:35.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:35.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:35.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:35.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:35.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:35.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:35.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:35.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:35.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:35.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:35.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:35.222+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:35.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:35.581+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.582521+0000) 2024-02-20T09:07:36.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:35.888+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.889393+0000) 2024-02-20T09:07:36.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:36.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:36.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:36.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:35.980+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:36.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:36.180+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56024 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:36.178+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.179802+0000) 2024-02-20T09:07:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:36.177+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.178428+0000) 2024-02-20T09:07:36.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:36.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.387625+0000) 2024-02-20T09:07:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:36.178+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.179230+0000) 2024-02-20T09:07:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:36.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:36.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:36.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:36.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:36.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:36.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:36.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:36.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:36.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:36.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:36.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:36.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:36.499+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:36.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:36.581+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.582708+0000) 2024-02-20T09:07:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:36.888+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.889568+0000) 2024-02-20T09:07:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:36.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:36.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:36.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:36.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:37.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:37.390+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:37.391036+0000) 2024-02-20T09:07:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:37.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:37.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:37.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:37.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:37.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:37.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:37.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:37.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:37.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:37.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:37.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:37.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:37.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:37.585+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:37.585958+0000) 2024-02-20T09:07:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:37.891+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:37.892865+0000) 2024-02-20T09:07:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:37.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:37.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:37.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:37.982+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:38.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:38.393+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:38.394463+0000) 2024-02-20T09:07:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:38.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:38.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:38.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:38.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:38.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:38.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:38.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:38.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:38.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:38.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:38.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:38.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:38.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:38.588+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:38.589210+0000) 2024-02-20T09:07:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:38.895+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:38.896289+0000) 2024-02-20T09:07:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:38.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:38.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:38.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:38.984+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:39.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:39.396+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:39.397645+0000) 2024-02-20T09:07:39.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:39.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:39.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:39.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:39.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:39.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:39.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:39.331+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:39.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:39.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:39.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:39.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:39.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:39.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:39.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:39.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:39.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:39.591+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:39.592421+0000) 2024-02-20T09:07:40.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:39.898+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:39.899509+0000) 2024-02-20T09:07:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:40.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:40.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:40.011+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:40.012+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:40.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:40.396+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.397842+0000) 2024-02-20T09:07:40.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:40.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:40.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:40.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:40.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:40.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:40.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:40.365+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:40.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:40.591+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.592617+0000) 2024-02-20T09:07:40.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:40.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:40.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:40.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:40.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:40.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:40.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:40.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:41.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:40.898+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.899711+0000) 2024-02-20T09:07:41.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:41.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:41.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:41.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:41.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:41.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:41.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:41.026+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:41.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:41.188+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56063 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:41.186+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.187018+0000) 2024-02-20T09:07:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:41.185+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.185976+0000) 2024-02-20T09:07:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:41.397+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.398028+0000) 2024-02-20T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:41.185+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.186449+0000) 2024-02-20T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:41.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:41.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:41.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:41.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:41.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:41.591+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.592727+0000) 2024-02-20T09:07:41.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:41.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:41.887 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:41.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:41.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:41.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:41.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:41.625+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:41.898+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.899894+0000) 2024-02-20T09:07:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:42.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:42.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:42.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:42.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:42.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:42.399+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:42.400359+0000) 2024-02-20T09:07:42.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:42.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:42.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:42.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:42.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:42.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:42.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:42.326+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:42.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:42.594+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:42.595402+0000) 2024-02-20T09:07:42.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:42.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:42.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:42.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:42.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:42.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:42.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:42.641+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:42.902+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:42.903337+0000) 2024-02-20T09:07:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:43.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:43.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:43.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:43.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:43.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:43.400+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:43.402820+0000) 2024-02-20T09:07:43.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:43.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:43.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:43.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:43.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:43.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:43.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:43.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:43.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:43.596+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:43.598769+0000) 2024-02-20T09:07:43.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:43.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:43.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:43.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:43.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:43.631+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:43.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:43.632+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:44.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:43.903+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:43.905891+0000) 2024-02-20T09:07:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:44.039+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:44.039+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:44.039+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:44.040+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:44.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:44.404+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:44.405928+0000) 2024-02-20T09:07:44.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:44.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:44.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:44.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:44.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:44.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:44.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:44.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:44.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:44.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:44.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:44.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:44.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:44.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:44.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:44.594+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:44.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:44.599+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:44.601156+0000) 2024-02-20T09:07:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:44.906+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:44.907974+0000) 2024-02-20T09:07:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:45.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:45.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:45.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:45.069+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:45.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:45.404+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.406108+0000) 2024-02-20T09:07:45.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:45.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:45.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:45.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:45.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:45.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:45.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:45.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:45.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:45.599+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.601338+0000) 2024-02-20T09:07:45.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:45.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:45.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:45.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:45.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:45.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:45.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:45.612+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:46.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:45.907+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.908192+0000) 2024-02-20T09:07:46.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:46.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:46.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:46.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:46.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:46.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:46.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:46.118+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:46.191+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.193792+0000) 2024-02-20T09:07:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:46.192+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.194256+0000) 2024-02-20T09:07:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:46.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:46.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:46.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:46.280+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:46.473 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:46.195+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56102 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:46.192+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.193997+0000) 2024-02-20T09:07:46.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:46.405+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.406290+0000) 2024-02-20T09:07:46.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:46.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:46.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:46.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:46.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:46.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:46.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:46.564+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:46.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:46.599+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.601440+0000) 2024-02-20T09:07:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:46.907+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.908461+0000) 2024-02-20T09:07:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:47.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:47.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:47.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:47.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:47.407+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:47.408825+0000) 2024-02-20T09:07:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:47.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:47.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:47.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:47.296+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:47.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:47.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:47.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:47.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:47.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:47.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:47.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:47.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:47.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:47.603+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:47.603945+0000) 2024-02-20T09:07:48.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:47.910+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:47.911720+0000) 2024-02-20T09:07:48.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:48.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:48.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:48.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:48.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:48.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:48.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:48.111+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:48.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:48.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:48.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:48.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:48.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:48.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:48.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:48.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:48.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:48.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:48.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:48.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:48.605+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:48.606653+0000) 2024-02-20T09:07:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:48.410+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:48.411263+0000) 2024-02-20T09:07:49.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:48.913+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:48.914268+0000) 2024-02-20T09:07:49.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:49.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:49.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:49.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:49.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:49.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:49.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:49.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:49.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:49.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:49.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:49.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:49.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:49.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:49.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:49.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:49.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:49.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:49.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:49.514+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:49.608+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:49.609198+0000) 2024-02-20T09:07:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:49.412+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:49.413919+0000) 2024-02-20T09:07:50.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:49.915+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:49.916558+0000) 2024-02-20T09:07:50.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:50.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:50.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:50.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:50.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:50.254+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:50.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:50.255+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:50.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:50.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:50.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:50.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:50.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:50.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:50.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:50.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:50.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:50.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:50.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:50.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:50.608+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.609406+0000) 2024-02-20T09:07:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:50.413+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.414141+0000) 2024-02-20T09:07:51.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:50.915+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.916742+0000) 2024-02-20T09:07:51.364 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:51.204+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56141 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:51.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:51.201+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.202440+0000) 2024-02-20T09:07:51.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:51.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:51.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:51.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:51.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:51.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:51.201+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.202296+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:51.203+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.204046+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:51.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:51.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:51.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:51.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:51.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:51.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:51.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:51.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:51.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:51.608+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.609715+0000) 2024-02-20T09:07:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:51.413+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.414392+0000) 2024-02-20T09:07:52.192 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:51.915+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.916924+0000) 2024-02-20T09:07:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:52.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:52.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:52.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:52.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:52.253+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:52.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:52.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:52.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:52.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:52.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:52.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:52.415+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:52.416914+0000) 2024-02-20T09:07:52.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:52.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:52.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:52.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:52.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:52.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:52.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:52.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:52.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:52.611+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:52.612016+0000) 2024-02-20T09:07:53.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:52.918+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:52.919155+0000) 2024-02-20T09:07:53.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:53.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:53.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:53.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:53.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:53.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:53.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:53.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:53.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:53.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:53.186+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:53.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:53.418+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:53.419390+0000) 2024-02-20T09:07:53.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:53.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:53.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:53.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:53.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:53.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:53.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:53.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:53.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:53.614+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:53.615472+0000) 2024-02-20T09:07:54.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:53.920+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:53.921717+0000) 2024-02-20T09:07:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:54.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:54.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:54.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:54.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:54.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:54.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:54.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:54.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:54.233+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:54.420+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:54.421791+0000) 2024-02-20T09:07:54.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:54.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:54.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:54.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:54.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:54.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:54.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:54.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:54.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:54.616+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:54.617605+0000) 2024-02-20T09:07:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:54.923+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:54.923957+0000) 2024-02-20T09:07:55.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:55.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:55.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:55.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:55.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:55.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:55.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:55.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:55.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:55.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:55.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:55.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:55.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:55.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:55.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:55.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:55.561 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:55.421+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.421967+0000) 2024-02-20T09:07:55.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:55.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:55.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:55.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:55.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:55.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:55.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:55.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:55.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:55.616+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.617770+0000) 2024-02-20T09:07:56.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:55.923+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.924134+0000) 2024-02-20T09:07:56.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:07:56.211+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56180 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:07:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:56.210+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.211126+0000) 2024-02-20T09:07:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:56.209+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.209998+0000) 2024-02-20T09:07:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:56.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:56.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:56.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:56.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:56.209+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.210103+0000) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:56.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:56.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:56.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:56.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:56.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:56.421+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.422154+0000) 2024-02-20T09:07:56.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:56.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:56.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:56.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:56.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:56.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:56.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:56.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:56.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:56.617+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.617934+0000) 2024-02-20T09:07:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:56.923+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.924341+0000) 2024-02-20T09:07:57.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:57.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:57.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:57.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:57.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:57.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:57.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:57.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:57.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:57.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:57.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:57.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:57.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:57.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:57.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:57.291+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:57.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:57.423+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:57.424738+0000) 2024-02-20T09:07:57.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:57.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:57.804 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:57.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:57.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:57.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:57.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:57.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:57.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:57.618+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:57.619882+0000) 2024-02-20T09:07:58.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:57.925+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:57.926409+0000) 2024-02-20T09:07:58.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:58.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:58.533 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:58.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:58.533 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:58.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:58.533 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:58.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:58.533 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:58.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:58.533 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:58.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:58.533 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:58.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:58.534 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:58.311+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:58.534 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:58.426+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:58.427659+0000) 2024-02-20T09:07:58.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:58.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:58.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:58.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:58.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:58.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:58.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:58.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:58.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:58.620+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:58.621895+0000) 2024-02-20T09:07:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:58.928+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:58.929047+0000) 2024-02-20T09:07:59.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:59.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:59.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:59.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:59.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:59.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:07:59.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:59.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:59.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:59.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:59.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:59.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:59.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:59.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:07:59.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:59.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:07:59.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:07:59.428+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:59.429708+0000) 2024-02-20T09:07:59.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:59.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:59.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:59.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:59.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:59.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:07:59.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:59.487+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:59.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:07:59.622+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:59.623771+0000) 2024-02-20T09:08:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:07:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:07:59.930+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:59.931094+0000) 2024-02-20T09:08:00.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:00.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:00.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:00.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:00.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:00.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:00.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:00.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:00.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:00.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:00.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:00.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:00.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:00.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:00.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:00.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:00.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:00.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:00.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:00.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:00.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:00.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:00.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:00.368+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:00.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:00.428+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.429880+0000) 2024-02-20T09:08:00.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:00.623+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.623953+0000) 2024-02-20T09:08:01.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:00.930+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.931266+0000) 2024-02-20T09:08:01.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:01.220+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56218 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:01.215+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.216924+0000) 2024-02-20T09:08:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:01.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:01.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:01.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:01.280+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:01.218+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.219152+0000) 2024-02-20T09:08:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:01.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:01.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:01.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:01.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:01.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:01.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:01.216+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.217405+0000) 2024-02-20T09:08:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:01.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:01.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:01.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:01.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:01.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:01.429+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.430069+0000) 2024-02-20T09:08:01.895 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:01.623+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.624136+0000) 2024-02-20T09:08:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:01.930+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.931462+0000) 2024-02-20T09:08:02.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:02.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:02.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:02.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:02.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:02.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:02.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:02.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:02.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:02.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:02.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:02.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:02.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:02.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:02.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:02.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:02.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:02.356+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:02.562 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:02.431+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:02.432098+0000) 2024-02-20T09:08:02.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:02.625+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:02.626213+0000) 2024-02-20T09:08:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:02.932+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:02.933887+0000) 2024-02-20T09:08:03.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:03.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:03.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:03.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:03.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:03.222+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:03.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:03.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:03.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:03.385+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:03.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:03.433+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:03.434249+0000) 2024-02-20T09:08:03.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:03.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:03.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:03.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:03.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:03.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:03.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:03.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:03.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:03.628+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:03.628968+0000) 2024-02-20T09:08:04.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:03.935+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:03.936516+0000) 2024-02-20T09:08:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:04.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:04.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:04.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:04.212+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:04.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:04.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:04.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:04.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:04.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:04.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:04.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:04.356+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:04.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:04.435+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:04.436837+0000) 2024-02-20T09:08:04.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:04.630+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:04.631488+0000) 2024-02-20T09:08:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:04.938+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:04.939838+0000) 2024-02-20T09:08:05.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:05.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:05.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:05.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:05.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:05.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:05.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:05.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:05.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:05.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:05.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:05.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:05.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:05.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:05.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:05.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:05.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:05.436+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.437025+0000) 2024-02-20T09:08:05.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:05.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:05.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:05.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:05.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:05.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:05.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:05.500+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:05.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:05.630+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.631681+0000) 2024-02-20T09:08:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:05.939+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.939973+0000) 2024-02-20T09:08:06.550 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:06.227+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56258 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:06.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:06.224+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.225930+0000) 2024-02-20T09:08:06.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:06.224+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.225330+0000) 2024-02-20T09:08:06.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:06.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:06.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:06.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:06.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:06.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:06.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:06.272+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:06.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:06.223+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.224796+0000) 2024-02-20T09:08:06.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:06.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:06.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:06.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:06.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:06.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:06.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:06.348+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:06.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:06.436+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.437192+0000) 2024-02-20T09:08:06.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:06.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:06.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:06.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:06.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:06.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:06.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:06.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:06.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:06.630+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.631866+0000) 2024-02-20T09:08:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:06.939+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.940159+0000) 2024-02-20T09:08:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:07.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:07.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:07.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:07.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:07.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:07.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:07.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:07.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:07.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:07.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:07.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:07.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:07.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:07.440+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:07.440972+0000) 2024-02-20T09:08:07.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:07.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:07.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:07.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:07.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:07.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:07.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:07.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:07.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:07.633+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:07.633964+0000) 2024-02-20T09:08:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:07.942+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:07.943548+0000) 2024-02-20T09:08:08.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:08.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:08.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:08.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:08.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:08.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:08.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:08.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:08.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:08.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:08.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:08.443+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:08.444275+0000) 2024-02-20T09:08:08.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:08.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:08.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:08.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:08.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:08.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:08.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:08.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:08.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:08.635+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:08.636369+0000) 2024-02-20T09:08:09.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:08.945+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:08.946201+0000) 2024-02-20T09:08:09.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:09.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:09.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:09.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:09.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:09.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:09.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:09.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:09.519 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:09.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:09.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:09.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:09.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:09.349+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:09.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:09.350+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:09.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:09.447+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:09.448826+0000) 2024-02-20T09:08:09.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:09.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:09.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:09.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:09.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:09.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:09.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:09.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:09.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:09.637+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:09.638621+0000) 2024-02-20T09:08:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:09.948+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:09.949354+0000) 2024-02-20T09:08:10.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:10.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:10.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:10.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:10.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:10.246+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:10.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:10.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:10.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:10.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:10.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:10.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:10.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:10.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:10.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:10.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:10.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:10.448+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.449029+0000) 2024-02-20T09:08:10.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:10.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:10.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:10.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:10.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:10.528+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:10.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:10.529+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:10.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:10.637+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.638822+0000) 2024-02-20T09:08:11.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:10.948+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.949552+0000) 2024-02-20T09:08:11.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:11.233+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56297 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:11.233+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.234322+0000) 2024-02-20T09:08:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:11.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:11.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:11.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:11.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:11.233+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.234245+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:11.230+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.231832+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:11.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:11.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:11.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:11.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:11.448+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.449229+0000) 2024-02-20T09:08:11.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:11.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:11.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:11.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:11.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:11.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:11.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:11.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:11.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:11.638+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.638955+0000) 2024-02-20T09:08:12.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:11.948+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.949792+0000) 2024-02-20T09:08:12.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:12.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:12.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:12.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:12.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:12.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:12.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:12.186+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:12.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:12.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:12.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:12.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:12.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:12.357+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:12.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:12.357+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:12.450+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:12.451409+0000) 2024-02-20T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:12.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:12.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:12.558+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:12.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:12.641+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:12.642148+0000) 2024-02-20T09:08:13.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:12.950+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:12.951409+0000) 2024-02-20T09:08:13.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:13.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:13.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:13.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:13.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:13.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:13.173+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:13.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:13.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:13.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:13.351+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:13.452+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:13.453759+0000) 2024-02-20T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:13.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:13.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:13.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:13.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:13.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:13.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:13.643+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:13.644314+0000) 2024-02-20T09:08:14.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:13.952+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:13.953415+0000) 2024-02-20T09:08:14.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:14.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:14.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:14.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:14.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:14.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:14.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:14.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:14.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:14.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:14.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:14.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:14.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:14.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:14.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:14.387+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:14.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:14.455+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:14.455970+0000) 2024-02-20T09:08:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:14.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:14.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:14.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:14.496+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:14.646+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:14.647450+0000) 2024-02-20T09:08:15.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:14.955+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:14.956026+0000) 2024-02-20T09:08:15.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:15.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:15.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:15.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:15.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:15.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:15.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:15.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:15.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:15.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:15.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:15.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:15.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:15.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:15.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:15.368+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:15.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:15.455+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.456161+0000) 2024-02-20T09:08:15.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:15.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:15.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:15.529+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:15.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:15.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:15.646+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.647646+0000) 2024-02-20T09:08:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:15.955+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.956192+0000) 2024-02-20T09:08:16.509 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:16.241+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56337 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:16.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:16.240+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.240970+0000) 2024-02-20T09:08:16.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:16.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:16.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:16.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:16.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:16.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:16.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:16.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:16.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:16.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:16.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:16.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:16.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:16.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:16.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:16.238+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.239228+0000) 2024-02-20T09:08:16.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:16.238+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.239815+0000) 2024-02-20T09:08:16.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:16.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:16.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:16.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:16.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:16.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:16.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:16.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:16.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:16.455+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.456362+0000) 2024-02-20T09:08:16.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:16.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:16.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:16.646+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.647818+0000) 2024-02-20T09:08:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:16.955+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.956359+0000) 2024-02-20T09:08:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:17.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:17.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:17.234+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:17.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:17.235+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:17.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:17.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:17.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:17.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:17.459+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:17.459934+0000) 2024-02-20T09:08:17.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:17.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:17.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:17.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:17.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:17.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:17.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:17.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:17.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:17.649+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:17.650270+0000) 2024-02-20T09:08:18.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:17.958+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:17.959695+0000) 2024-02-20T09:08:18.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:18.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:18.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:18.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:18.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:18.241+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:18.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:18.241+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:18.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:18.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:18.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:18.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:18.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:18.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:18.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:18.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:18.555 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:18.461+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:18.462470+0000) 2024-02-20T09:08:18.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:18.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:18.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:18.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:18.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:18.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:18.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:18.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:18.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:18.652+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:18.653631+0000) 2024-02-20T09:08:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:18.962+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:18.963147+0000) 2024-02-20T09:08:19.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:19.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:19.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:19.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:19.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:19.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:19.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:19.243+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:19.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:19.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:19.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:19.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:19.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:19.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:19.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:19.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:19.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:19.463+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:19.464795+0000) 2024-02-20T09:08:19.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:19.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:19.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:19.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:19.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:19.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:19.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:19.536+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:19.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:19.655+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:19.656167+0000) 2024-02-20T09:08:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:19.964+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:19.965657+0000) 2024-02-20T09:08:20.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:20.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:20.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:20.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:20.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:20.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:20.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:20.231+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:20.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:20.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:20.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:20.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:20.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:20.464+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.464977+0000) 2024-02-20T09:08:20.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:20.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:20.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:20.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:20.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:20.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:20.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:20.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:20.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:20.655+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.656356+0000) 2024-02-20T09:08:21.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:20.964+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.965860+0000) 2024-02-20T09:08:21.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:21.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:21.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:21.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:21.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:21.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:21.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:21.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:21.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:21.248+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.249269+0000) 2024-02-20T09:08:21.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:21.245+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.246903+0000) 2024-02-20T09:08:21.467 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:21.248+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56376 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:21.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:21.245+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.246521+0000) 2024-02-20T09:08:21.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:21.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:21.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:21.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:21.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:21.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:21.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:21.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:21.464+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.465168+0000) 2024-02-20T09:08:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:21.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:21.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:21.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:21.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:21.655+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.656525+0000) 2024-02-20T09:08:22.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:21.965+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.966016+0000) 2024-02-20T09:08:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:22.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:22.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:22.194+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:22.195+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:22.470 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:22.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:22.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:22.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:22.288+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:22.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:22.467+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:22.468820+0000) 2024-02-20T09:08:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:22.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:22.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:22.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:22.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:22.657+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:22.658815+0000) 2024-02-20T09:08:23.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:22.967+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:22.968029+0000) 2024-02-20T09:08:23.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:23.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:23.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:23.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:23.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:23.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:23.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:23.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:23.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:23.285+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:23.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:23.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:23.470+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:23.471563+0000) 2024-02-20T09:08:23.779 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:23.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:23.779 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:23.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:23.779 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:23.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:23.779 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:23.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:23.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:23.660+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:23.660940+0000) 2024-02-20T09:08:24.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:23.970+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:23.971400+0000) 2024-02-20T09:08:24.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:24.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:24.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:24.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:24.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:24.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:24.475 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:24.266+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:24.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:24.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:24.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:24.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:24.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:24.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:24.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:24.220+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:24.807 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:24.472+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:24.473578+0000) 2024-02-20T09:08:24.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:24.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:24.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:24.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:24.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:24.522+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:24.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:24.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:24.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:24.662+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:24.663411+0000) 2024-02-20T09:08:25.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:24.972+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:24.973872+0000) 2024-02-20T09:08:25.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:25.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:25.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:25.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:25.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:25.211+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:25.212+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:25.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:25.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:25.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:25.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:25.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:25.807 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:25.472+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.473784+0000) 2024-02-20T09:08:25.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:25.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:25.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:25.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:25.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:25.571+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:25.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:25.572+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:25.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:25.662+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.663594+0000) 2024-02-20T09:08:26.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:25.973+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.974058+0000) 2024-02-20T09:08:26.563 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:26.255+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56416 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:26.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:26.251+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.252900+0000) 2024-02-20T09:08:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:26.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:26.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:26.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:26.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:26.254+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.254970+0000) 2024-02-20T09:08:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:26.252+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.253351+0000) 2024-02-20T09:08:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:26.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:26.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:26.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:26.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:26.473+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.473957+0000) 2024-02-20T09:08:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:26.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:26.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:26.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:26.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:26.662+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.663792+0000) 2024-02-20T09:08:27.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:26.973+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.974227+0000) 2024-02-20T09:08:27.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:27.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:27.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:27.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:27.227+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:27.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:27.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:27.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:27.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:27.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:27.475+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:27.476363+0000) 2024-02-20T09:08:27.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:27.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:27.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:27.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:27.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:27.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:27.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:27.657+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:27.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:27.666+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:27.667300+0000) 2024-02-20T09:08:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:27.976+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:27.977580+0000) 2024-02-20T09:08:28.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:28.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:28.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:28.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:28.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:28.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:28.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:28.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:28.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:28.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:28.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:28.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:28.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:28.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:28.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:28.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:28.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:28.478+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:28.479548+0000) 2024-02-20T09:08:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:28.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:28.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:28.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:28.638+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:28.668+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:28.669440+0000) 2024-02-20T09:08:29.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:28.980+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:28.981013+0000) 2024-02-20T09:08:29.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:29.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:29.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:29.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:29.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:29.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:29.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:29.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:29.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:29.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:29.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:29.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:29.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:29.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:29.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:29.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:29.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:29.481+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:29.482088+0000) 2024-02-20T09:08:29.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:29.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:29.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:29.605+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:29.606+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:29.670+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:29.671583+0000) 2024-02-20T09:08:30.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:29.982+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:29.983664+0000) 2024-02-20T09:08:30.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:30.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:30.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:30.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:30.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:30.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:30.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:30.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:30.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:30.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:30.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:30.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:30.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:30.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:30.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:30.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:30.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:30.481+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.482291+0000) 2024-02-20T09:08:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:30.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:30.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:30.629+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:30.630+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:30.670+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.671770+0000) 2024-02-20T09:08:31.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:30.982+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.983838+0000) 2024-02-20T09:08:31.563 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:31.263+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56455 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:31.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:31.260+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.261538+0000) 2024-02-20T09:08:31.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:31.260+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.261483+0000) 2024-02-20T09:08:31.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:31.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:31.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:31.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:31.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:31.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:31.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:31.293+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:31.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:31.260+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.261113+0000) 2024-02-20T09:08:31.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:31.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:31.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:31.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:31.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:31.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:31.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:31.327+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:31.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:31.481+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.482488+0000) 2024-02-20T09:08:31.898 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:31.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:31.898 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:31.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:31.898 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:31.647+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:31.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:31.649+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:31.899 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:31.671+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.671964+0000) 2024-02-20T09:08:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:31.983+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.984010+0000) 2024-02-20T09:08:32.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:32.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:32.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:32.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:32.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:32.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:32.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:32.331+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:32.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:32.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:32.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:32.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:32.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:32.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:32.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:32.292+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:32.808 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:32.485+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:32.486131+0000) 2024-02-20T09:08:32.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:32.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:32.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:32.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:32.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:32.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:32.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:32.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:32.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:32.673+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:32.674035+0000) 2024-02-20T09:08:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:32.985+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:32.985951+0000) 2024-02-20T09:08:33.490 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:33.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:33.491 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:33.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:33.491 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:33.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:33.491 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:33.371+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:33.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:33.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:33.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:33.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:33.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:33.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:33.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:33.340+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:33.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:33.488+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:33.489404+0000) 2024-02-20T09:08:33.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:33.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:33.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:33.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:33.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:33.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:33.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:33.659+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:33.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:33.676+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:33.677153+0000) 2024-02-20T09:08:34.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:33.988+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:33.989224+0000) 2024-02-20T09:08:34.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:34.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:34.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:34.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:34.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:34.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:34.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:34.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:34.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:34.491+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:34.492317+0000) 2024-02-20T09:08:34.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:34.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:34.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:34.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:34.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:34.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:34.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:34.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:34.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:34.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:34.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:34.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:34.678+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:34.679656+0000) 2024-02-20T09:08:35.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:34.990+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:34.991873+0000) 2024-02-20T09:08:35.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:35.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:35.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:35.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:35.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:35.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:35.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:35.444+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:35.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:35.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:35.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:35.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:35.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:35.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:35.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:35.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:35.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:35.491+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.492520+0000) 2024-02-20T09:08:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:35.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:35.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:35.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:35.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:35.678+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.679852+0000) 2024-02-20T09:08:36.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:35.991+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.992072+0000) 2024-02-20T09:08:36.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:36.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:36.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:36.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:36.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:36.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:36.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:36.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:36.564 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:36.270+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56495 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:36.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:36.268+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.269594+0000) 2024-02-20T09:08:36.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:36.267+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.268631+0000) 2024-02-20T09:08:36.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:36.490+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.492692+0000) 2024-02-20T09:08:36.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:36.266+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.267818+0000) 2024-02-20T09:08:36.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:36.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:36.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:36.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:36.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:36.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:36.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:36.486+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:36.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:36.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:36.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:36.644+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:36.678+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.680034+0000) 2024-02-20T09:08:37.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:36.990+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.992253+0000) 2024-02-20T09:08:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:37.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:37.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:37.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:37.463+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:37.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:37.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:37.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:37.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:37.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:37.492+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:37.494694+0000) 2024-02-20T09:08:37.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:37.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:37.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:37.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:37.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:37.637+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:37.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:37.639+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:37.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:37.680+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:37.682143+0000) 2024-02-20T09:08:38.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:37.993+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:37.995764+0000) 2024-02-20T09:08:38.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:38.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:38.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:38.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:38.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:38.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:38.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:38.470+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:38.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:38.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:38.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:38.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:38.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:38.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:38.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:38.291+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:38.808 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:38.495+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:38.497509+0000) 2024-02-20T09:08:38.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:38.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:38.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:38.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:38.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:38.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:38.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:38.657+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:38.809 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:38.682+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:38.684651+0000) 2024-02-20T09:08:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:38.997+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:38.999320+0000) 2024-02-20T09:08:39.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:39.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:39.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:39.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:39.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:39.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:39.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:39.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:39.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:39.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:39.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:39.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:39.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:39.500+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:39.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:39.501+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:39.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:39.499+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:39.500073+0000) 2024-02-20T09:08:39.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:39.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:39.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:39.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:39.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:39.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:39.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:39.648+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:39.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:39.686+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:39.688132+0000) 2024-02-20T09:08:40.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:40.000+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.001913+0000) 2024-02-20T09:08:40.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:40.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:40.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:40.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:40.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:40.335+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:40.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:40.336+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:40.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:40.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:40.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:40.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:40.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:40.535+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:40.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:40.536+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:40.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:40.499+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.500374+0000) 2024-02-20T09:08:40.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:40.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:40.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:40.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:40.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:40.613+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:40.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:40.614+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:40.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:40.687+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.688285+0000) 2024-02-20T09:08:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:41.001+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.002103+0000) 2024-02-20T09:08:41.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:41.274+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.275682+0000) 2024-02-20T09:08:41.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:41.274+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.275721+0000) 2024-02-20T09:08:41.503 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:41.277+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56534 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:41.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:41.275+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.276292+0000) 2024-02-20T09:08:41.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:41.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:41.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:41.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:41.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:41.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:41.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:41.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:41.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:41.499+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.500682+0000) 2024-02-20T09:08:41.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:41.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:41.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:41.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:41.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:41.530+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:41.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:41.531+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:41.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:41.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:41.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:41.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:41.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:41.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:41.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:41.662+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:41.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:41.687+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.688466+0000) 2024-02-20T09:08:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:42.001+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:42.002315+0000) 2024-02-20T09:08:42.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:42.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:42.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:42.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:42.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:42.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:42.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:42.377+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:42.809 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:42.502+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:42.503314+0000) 2024-02-20T09:08:42.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:42.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:42.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:42.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:42.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:42.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:42.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:42.555+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:42.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:42.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:42.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:42.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:42.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:42.644+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:42.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:42.645+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:42.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:42.690+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:42.691053+0000) 2024-02-20T09:08:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:43.004+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:43.005905+0000) 2024-02-20T09:08:43.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:43.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:43.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:43.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:43.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:43.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:43.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:43.384+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:43.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:43.504+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:43.505802+0000) 2024-02-20T09:08:43.780 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:43.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:43.780 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:43.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:43.780 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:43.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:43.780 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:43.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:43.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:43.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:43.780 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:43.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:43.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:43.685+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:43.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:43.686+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:43.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:43.733+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:43.734725+0000) 2024-02-20T09:08:44.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:44.007+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:44.008563+0000) 2024-02-20T09:08:44.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:44.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:44.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:44.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:44.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:44.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:44.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:44.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:44.809 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:44.507+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:44.508028+0000) 2024-02-20T09:08:44.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:44.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:44.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:44.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:44.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:44.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:44.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:44.506+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:44.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:44.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:44.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:44.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:44.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:44.663+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:44.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:44.664+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:44.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:44.737+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:44.738187+0000) 2024-02-20T09:08:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:45.011+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.012108+0000) 2024-02-20T09:08:45.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:45.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:45.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:45.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:45.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:45.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:45.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:45.390+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:45.809 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:45.507+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.508221+0000) 2024-02-20T09:08:45.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:45.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:45.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:45.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:45.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:45.536+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:45.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:45.537+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:45.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:45.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:45.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:45.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:45.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:45.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:45.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:45.695+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:45.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:45.737+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.738389+0000) 2024-02-20T09:08:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:46.011+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.012306+0000) 2024-02-20T09:08:46.500 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:46.284+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56575 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:46.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:46.282+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.283054+0000) 2024-02-20T09:08:46.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:46.282+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.283463+0000) 2024-02-20T09:08:46.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:46.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:46.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:46.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:46.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:46.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:46.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:46.283+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.284580+0000) 2024-02-20T09:08:46.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:46.378+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:46.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:46.378+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:46.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:46.378+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:46.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:46.378+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:46.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:46.499+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:46.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:46.507+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.508409+0000) 2024-02-20T09:08:46.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:46.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:46.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:46.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:46.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:46.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:46.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:46.667+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:46.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:46.737+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.738552+0000) 2024-02-20T09:08:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:47.011+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:47.012480+0000) 2024-02-20T09:08:47.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:47.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:47.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:47.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:47.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:47.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:47.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:47.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:47.809 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:47.509+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:47.510757+0000) 2024-02-20T09:08:47.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:47.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:47.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:47.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:47.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:47.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:47.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:47.506+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:47.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:47.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:47.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:47.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:47.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:47.698+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:47.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:47.699+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:47.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:47.739+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:47.740604+0000) 2024-02-20T09:08:48.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:48.015+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:48.016012+0000) 2024-02-20T09:08:48.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:48.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:48.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:48.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:48.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:48.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:48.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:48.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:48.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:48.512+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:48.513382+0000) 2024-02-20T09:08:48.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:48.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:48.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:48.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:48.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:48.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:48.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:48.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:48.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:48.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:48.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:48.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:48.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:48.697+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:48.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:48.697+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:48.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:48.741+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:48.742681+0000) 2024-02-20T09:08:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:49.018+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:49.019551+0000) 2024-02-20T09:08:49.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:49.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:49.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:49.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:49.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:49.547+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:49.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:49.547+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:49.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:49.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:49.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:49.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:49.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:49.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:49.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:49.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:49.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:49.515+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:49.516115+0000) 2024-02-20T09:08:49.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:49.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:49.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:49.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:49.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:49.669+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:49.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:49.669+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:49.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:49.744+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:49.745177+0000) 2024-02-20T09:08:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:50.021+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.022093+0000) 2024-02-20T09:08:50.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:50.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:50.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:50.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:50.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:50.569+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:50.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:50.570+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:50.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:50.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:50.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:50.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:50.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:50.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:50.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:50.692+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:50.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:50.744+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.745378+0000) 2024-02-20T09:08:50.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:50.515+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.516281+0000) 2024-02-20T09:08:50.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:50.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:50.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:50.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:50.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:50.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:50.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:50.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:51.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:51.021+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.022243+0000) 2024-02-20T09:08:51.517 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:51.292+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56614 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:51.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:51.289+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.290901+0000) 2024-02-20T09:08:51.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:51.290+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.290979+0000) 2024-02-20T09:08:51.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:51.288+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.289882+0000) 2024-02-20T09:08:51.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:51.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:51.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:51.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:51.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:51.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:51.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:51.533+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:51.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:51.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:51.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:51.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:51.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:51.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:51.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:51.696+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:51.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:51.744+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.745576+0000) 2024-02-20T09:08:51.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:51.515+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.516379+0000) 2024-02-20T09:08:51.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:51.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:51.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:51.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:51.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:51.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:51.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:51.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:52.021+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:52.022419+0000) 2024-02-20T09:08:52.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:52.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:52.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:52.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:52.810 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:52.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:52.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:52.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:52.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:52.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:52.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:52.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:52.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:52.679+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:52.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:52.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:52.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:52.746+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:52.747538+0000) 2024-02-20T09:08:52.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:52.518+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:52.519588+0000) 2024-02-20T09:08:52.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:52.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:52.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:52.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:52.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:52.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:52.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:52.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:53.023+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:53.024912+0000) 2024-02-20T09:08:53.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:53.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:53.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:53.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:53.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:53.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:53.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:53.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:53.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:53.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:53.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:53.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:53.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:53.690+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:53.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:53.691+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:53.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:53.750+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:53.751118+0000) 2024-02-20T09:08:53.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:53.522+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:53.523160+0000) 2024-02-20T09:08:53.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:53.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:53.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:53.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:53.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:53.584+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:53.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:53.584+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:54.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:54.026+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:54.027516+0000) 2024-02-20T09:08:54.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:54.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:54.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:54.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:54.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:54.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:54.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:54.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:54.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:54.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:54.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:54.653+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:54.654+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:54.752+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:54.753605+0000) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:54.524+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:54.525777+0000) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:54.543+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:54.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:54.543+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:54.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:54.543+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:54.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:54.544+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:55.030+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.031034+0000) 2024-02-20T09:08:55.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:55.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:55.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:55.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:55.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:55.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:55.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:55.487+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:55.525+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.525980+0000) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:55.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:55.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:55.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:55.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:55.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:55.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:55.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:55.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:55.684+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:55.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:55.685+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:55.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:55.752+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.753802+0000) 2024-02-20T09:08:56.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:56.030+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.031228+0000) 2024-02-20T09:08:56.566 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:08:56.300+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56654 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:08:56.567 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:56.297+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.298699+0000) 2024-02-20T09:08:56.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:56.297+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.298459+0000) 2024-02-20T09:08:56.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:56.525+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.526134+0000) 2024-02-20T09:08:56.567 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:56.296+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.297529+0000) 2024-02-20T09:08:56.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:56.533+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:56.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:56.533+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:56.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:56.533+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:56.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:56.534+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:56.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:56.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:56.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:56.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:56.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:56.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:56.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:56.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:56.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:56.753+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.753971+0000) 2024-02-20T09:08:57.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:57.030+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:57.031416+0000) 2024-02-20T09:08:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:57.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:57.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:57.534+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:57.535+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:57.528+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:57.529422+0000) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:57.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:57.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:57.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:57.546+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:58.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:57.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:58.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:57.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:58.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:57.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:58.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:57.749+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:58.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:57.756+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:57.757515+0000) 2024-02-20T09:08:58.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:58.033+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:58.034768+0000) 2024-02-20T09:08:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:58.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:58.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:58.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:58.531+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:58.532116+0000) 2024-02-20T09:08:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:58.532+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:58.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:58.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:58.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:58.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:58.519+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:59.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:58.759+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:58.760066+0000) 2024-02-20T09:08:59.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:58.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:59.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:58.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:59.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:58.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:08:59.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:58.763+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:59.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:59.036+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:59.037898+0000) 2024-02-20T09:08:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:59.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:59.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:59.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:08:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:08:59.475+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:08:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:59.535+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:59.535942+0000) 2024-02-20T09:08:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:59.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:59.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:59.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:08:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:08:59.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:00.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:59.761+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:59.762533+0000) 2024-02-20T09:09:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:59.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:59.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:59.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:08:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:08:59.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:00.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:00.039+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.040552+0000) 2024-02-20T09:09:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:00.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:00.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:00.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:00.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:00.535+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.536121+0000) 2024-02-20T09:09:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:00.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:00.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:00.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:00.576+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:00.761+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.762721+0000) 2024-02-20T09:09:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:00.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:00.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:00.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:00.779+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:01.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:01.039+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.040709+0000) 2024-02-20T09:09:01.566 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:01.305+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.306101+0000) 2024-02-20T09:09:01.567 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:01.307+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56693 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:01.567 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:01.304+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.305559+0000) 2024-02-20T09:09:01.567 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:01.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:01.567 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:01.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:01.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:01.446+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:01.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:01.446+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:01.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:01.305+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.306554+0000) 2024-02-20T09:09:01.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:01.535+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.536278+0000) 2024-02-20T09:09:01.900 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:01.761+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.762911+0000) 2024-02-20T09:09:01.900 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:01.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:01.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:01.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:01.798+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:01.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:01.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:01.571+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:01.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:01.572+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:02.039+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:02.040900+0000) 2024-02-20T09:09:02.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:02.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:02.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:02.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:02.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:02.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:02.764+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:02.765428+0000) 2024-02-20T09:09:02.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:02.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:02.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:02.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:02.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:02.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:02.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:02.779+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:02.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:02.538+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:02.539603+0000) 2024-02-20T09:09:02.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:02.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:02.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:02.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:02.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:02.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:02.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:02.569+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:03.042+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:03.043402+0000) 2024-02-20T09:09:03.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:03.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:03.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:03.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:03.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:03.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:03.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:03.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:03.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:03.767+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:03.768018+0000) 2024-02-20T09:09:03.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:03.541+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:03.542420+0000) 2024-02-20T09:09:03.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:03.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:03.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:03.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:03.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:03.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:03.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:03.605+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:04.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:04.045+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:04.046736+0000) 2024-02-20T09:09:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:03.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:03.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:03.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:03.810+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:04.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:04.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:04.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:04.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:04.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:04.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:04.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:04.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:04.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:04.769+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:04.770431+0000) 2024-02-20T09:09:04.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:04.544+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:04.545089+0000) 2024-02-20T09:09:04.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:04.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:04.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:04.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:04.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:04.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:04.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:04.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:05.049+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.050037+0000) 2024-02-20T09:09:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:04.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:04.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:04.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:04.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:05.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:05.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:05.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:05.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:05.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:05.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:05.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:05.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:05.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:05.769+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.770604+0000) 2024-02-20T09:09:05.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:05.544+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.545287+0000) 2024-02-20T09:09:05.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:05.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:05.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:05.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:05.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:05.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:05.813 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:05.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:06.049+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.050214+0000) 2024-02-20T09:09:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:05.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:05.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:05.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:05.836+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:06.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:06.312+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.313361+0000) 2024-02-20T09:09:06.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:06.311+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.312284+0000) 2024-02-20T09:09:06.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:06.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:06.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:06.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:06.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:06.464+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:06.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:06.465+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:06.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:06.314+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56733 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:06.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:06.312+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.313071+0000) 2024-02-20T09:09:06.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:06.769+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.770803+0000) 2024-02-20T09:09:06.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:06.544+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.545473+0000) 2024-02-20T09:09:06.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:06.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:06.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:06.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:06.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:06.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:06.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:06.698+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:07.049+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:07.050376+0000) 2024-02-20T09:09:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:06.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:06.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:06.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:06.878+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:07.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:07.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:07.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:07.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:07.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:07.547+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:07.548152+0000) 2024-02-20T09:09:07.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:07.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:07.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:07.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:07.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:07.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:07.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:07.659+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:08.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:07.772+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:07.772974+0000) 2024-02-20T09:09:08.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:07.837+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:08.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:07.837+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:08.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:07.837+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:08.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:07.838+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:08.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:08.051+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:08.052533+0000) 2024-02-20T09:09:08.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:08.549+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:08.550558+0000) 2024-02-20T09:09:08.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:08.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:08.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:08.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:08.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:08.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:08.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:08.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:08.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:08.451+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:08.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:08.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:08.614+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:08.774+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:08.775827+0000) 2024-02-20T09:09:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:08.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:08.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:08.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:08.861+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:09.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:09.054+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:09.055441+0000) 2024-02-20T09:09:09.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:09.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:09.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:09.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:09.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:09.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:09.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:09.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:09.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:09.553+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:09.553942+0000) 2024-02-20T09:09:09.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:09.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:09.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:09.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:09.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:09.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:09.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:09.777+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:09.778278+0000) 2024-02-20T09:09:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:09.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:09.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:09.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:09.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:10.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:10.057+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.058036+0000) 2024-02-20T09:09:10.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:10.553+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.554117+0000) 2024-02-20T09:09:10.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:10.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:10.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:10.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:10.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:10.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:10.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:10.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:10.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:10.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:10.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:10.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:10.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:10.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:10.777+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.778454+0000) 2024-02-20T09:09:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:10.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:10.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:10.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:10.798+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:11.239 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:11.057+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.058231+0000) 2024-02-20T09:09:11.494 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:11.322+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56772 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:11.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:11.319+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.319916+0000) 2024-02-20T09:09:11.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:11.320+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.321304+0000) 2024-02-20T09:09:11.495 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:11.318+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.319820+0000) 2024-02-20T09:09:11.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:11.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:11.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:11.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:11.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:11.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:11.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:11.493+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:11.778 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:11.553+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.554310+0000) 2024-02-20T09:09:11.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:11.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:11.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:11.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:11.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:11.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:11.779 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:11.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:12.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:11.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:12.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:11.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:12.060 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:11.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:12.060 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:11.776+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:12.060 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:11.777+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.778641+0000) 2024-02-20T09:09:12.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:12.057+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:12.058429+0000) 2024-02-20T09:09:12.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:12.555+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:12.556641+0000) 2024-02-20T09:09:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:12.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:12.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:12.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:12.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:13.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:12.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:13.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:12.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:13.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:12.743+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:13.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:12.744+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:13.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:12.779+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:12.780626+0000) 2024-02-20T09:09:13.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:12.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:13.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:12.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:13.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:12.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:13.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:12.736+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:13.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:13.059+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:13.060836+0000) 2024-02-20T09:09:13.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:13.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:13.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:13.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:13.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:13.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:13.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:13.453+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:13.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:13.558+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:13.559888+0000) 2024-02-20T09:09:13.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:13.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:13.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:13.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:13.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:13.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:13.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:13.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:13.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:13.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:13.770+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:13.781+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:13.782779+0000) 2024-02-20T09:09:14.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:14.063+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:14.064343+0000) 2024-02-20T09:09:14.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:14.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:14.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:14.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:14.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:14.462+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:14.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:14.463+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:14.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:14.562+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:14.562975+0000) 2024-02-20T09:09:14.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:14.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:14.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:14.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:14.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:14.648+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:14.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:14.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:14.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:14.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:14.777+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:14.783+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:14.784852+0000) 2024-02-20T09:09:15.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:15.066+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.067679+0000) 2024-02-20T09:09:15.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:15.562+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.563152+0000) 2024-02-20T09:09:15.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:15.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:15.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:15.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:15.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:15.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:15.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:15.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:15.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:15.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:15.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:15.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:15.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:15.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:15.784+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.785041+0000) 2024-02-20T09:09:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:15.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:15.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:15.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:15.806+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:16.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:16.066+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.067881+0000) 2024-02-20T09:09:16.704 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:16.329+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56812 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:16.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:16.326+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.327752+0000) 2024-02-20T09:09:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:16.327+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.328169+0000) 2024-02-20T09:09:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:16.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:16.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:16.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:16.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:16.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:16.327+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.328252+0000) 2024-02-20T09:09:16.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:16.562+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.563336+0000) 2024-02-20T09:09:16.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:16.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:16.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:16.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:16.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:16.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:16.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:16.784+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.785132+0000) 2024-02-20T09:09:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:16.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:16.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:16.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:16.843+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:17.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:17.067+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:17.068082+0000) 2024-02-20T09:09:17.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:17.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:17.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:17.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:17.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:17.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:17.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:17.495+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:17.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:17.565+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:17.566618+0000) 2024-02-20T09:09:17.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:17.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:17.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:17.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:17.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:17.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:17.702+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:17.786+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:17.787712+0000) 2024-02-20T09:09:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:17.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:17.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:17.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:17.855+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:18.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:18.069+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:18.070427+0000) 2024-02-20T09:09:18.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:18.568+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:18.569810+0000) 2024-02-20T09:09:18.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:18.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:18.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:18.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:18.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:18.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:18.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:18.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:18.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:18.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:18.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:18.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:18.788+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:18.789747+0000) 2024-02-20T09:09:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:18.836+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:18.836+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:18.836+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:18.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:18.837+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:19.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:19.073+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:19.073968+0000) 2024-02-20T09:09:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:19.571+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:19.572491+0000) 2024-02-20T09:09:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:19.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:19.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:19.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:19.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:19.515+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:19.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:19.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:19.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:19.725+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:19.791+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:19.792259+0000) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:19.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:20.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:19.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:20.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:19.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:20.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:19.849+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:20.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:20.076+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.077558+0000) 2024-02-20T09:09:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:20.571+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.572690+0000) 2024-02-20T09:09:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:20.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:20.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:20.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:20.538+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:21.079 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:20.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:21.079 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:20.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:21.080 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:20.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:21.080 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:20.727+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:21.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:20.791+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.792444+0000) 2024-02-20T09:09:21.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:20.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:21.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:20.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:21.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:20.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:21.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:20.815+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:21.332 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:21.077+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.078095+0000) 2024-02-20T09:09:21.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:21.333+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.334539+0000) 2024-02-20T09:09:21.708 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:21.336+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56851 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:21.708 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:21.334+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.334959+0000) 2024-02-20T09:09:21.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:21.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:21.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:21.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:21.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:21.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:21.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:21.542+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:21.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:21.334+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.335564+0000) 2024-02-20T09:09:21.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:21.571+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.572852+0000) 2024-02-20T09:09:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:21.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:21.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:21.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:21.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:21.791+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.792601+0000) 2024-02-20T09:09:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:21.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:21.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:21.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:21.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:21.794+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:22.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:22.077+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:22.078279+0000) 2024-02-20T09:09:22.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:22.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:22.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:22.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:22.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:22.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:22.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:22.540+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:22.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:22.574+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:22.575286+0000) 2024-02-20T09:09:22.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:22.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:22.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:22.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:22.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:22.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:22.709+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:22.794+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:22.795489+0000) 2024-02-20T09:09:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:22.819+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:22.819+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:22.819+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:22.820+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:23.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:23.079+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:23.080805+0000) 2024-02-20T09:09:23.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:23.578+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:23.578933+0000) 2024-02-20T09:09:23.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:23.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:23.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:23.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:23.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:23.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:23.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:23.517+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:23.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:23.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:23.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:23.698+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:23.797+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:23.797977+0000) 2024-02-20T09:09:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:23.866+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:23.866+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:23.866+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:23.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:23.866+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:24.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:24.082+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:24.083357+0000) 2024-02-20T09:09:24.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:24.580+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:24.581378+0000) 2024-02-20T09:09:24.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:24.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:24.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:24.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:24.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:24.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:24.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:24.499+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:24.800+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:24.801601+0000) 2024-02-20T09:09:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:24.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:24.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:24.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:24.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:24.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:24.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:24.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:24.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:25.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:25.084+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.085790+0000) 2024-02-20T09:09:25.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:25.580+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.581595+0000) 2024-02-20T09:09:25.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:25.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:25.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:25.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:25.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:25.691+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:25.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:25.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:25.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:25.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:25.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:25.456+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:25.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:25.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:25.692+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:25.800+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.801846+0000) 2024-02-20T09:09:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:25.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:25.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:25.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:25.815+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:26.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:26.085+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.086175+0000) 2024-02-20T09:09:26.673 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:26.343+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56891 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:26.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:26.340+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.341335+0000) 2024-02-20T09:09:26.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:26.342+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.343077+0000) 2024-02-20T09:09:26.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:26.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:26.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:26.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:26.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:26.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:26.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:26.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:26.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:26.341+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.342929+0000) 2024-02-20T09:09:26.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:26.580+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.581873+0000) 2024-02-20T09:09:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:26.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:26.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:26.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:26.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:26.801+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.802053+0000) 2024-02-20T09:09:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:26.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:26.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:26.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:26.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:26.824+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:27.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:27.085+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:27.086368+0000) 2024-02-20T09:09:27.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:27.583+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:27.584419+0000) 2024-02-20T09:09:27.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:27.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:27.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:27.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:27.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:27.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:27.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:27.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:27.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:27.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:27.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:27.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:27.803+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:27.804737+0000) 2024-02-20T09:09:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:27.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:27.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:27.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:27.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:28.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:28.088+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:28.088950+0000) 2024-02-20T09:09:28.618 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:28.586+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:28.587090+0000) 2024-02-20T09:09:28.619 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:28.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:28.619 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:28.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:28.619 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:28.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:28.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:28.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:28.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:28.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:28.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:28.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:28.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:28.461+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:28.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:28.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:28.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:28.777+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:28.806+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:28.807452+0000) 2024-02-20T09:09:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:28.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:29.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:29.091+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:29.092396+0000) 2024-02-20T09:09:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:29.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:29.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:29.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:29.452+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:29.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:29.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:29.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:29.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:29.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:29.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:29.588+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:29.589507+0000) 2024-02-20T09:09:30.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:29.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:30.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:29.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:30.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:29.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:30.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:29.806+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:30.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:29.807+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:29.809637+0000) 2024-02-20T09:09:30.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:30.093+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.095877+0000) 2024-02-20T09:09:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:30.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:30.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:30.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:30.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:30.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:30.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:30.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:30.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:30.539+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:30.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:30.587+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.589763+0000) 2024-02-20T09:09:31.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:30.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:31.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:30.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:31.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:30.762+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:31.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:30.763+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:31.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:30.807+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.809878+0000) 2024-02-20T09:09:31.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:31.094+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.096132+0000) 2024-02-20T09:09:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:31.349+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56930 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:31.348+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.350657+0000) 2024-02-20T09:09:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:31.348+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.350860+0000) 2024-02-20T09:09:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:31.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:31.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:31.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:31.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:31.346+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.348681+0000) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:31.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:31.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:31.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:31.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:31.519+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:31.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:31.587+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.589945+0000) 2024-02-20T09:09:32.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:31.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:32.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:31.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:32.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:31.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:32.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:31.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:32.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:31.808+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.810014+0000) 2024-02-20T09:09:32.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:32.094+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:32.096303+0000) 2024-02-20T09:09:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:32.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:32.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:32.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:32.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:32.386+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:32.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:32.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:32.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:32.562+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:32.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:32.591+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:32.592511+0000) 2024-02-20T09:09:33.100 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:32.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:33.100 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:32.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:33.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:32.788+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:33.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:32.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:33.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:32.812+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:32.813376+0000) 2024-02-20T09:09:33.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:33.098+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:33.098995+0000) 2024-02-20T09:09:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:33.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:33.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:33.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:33.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:33.593+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:33.594771+0000) 2024-02-20T09:09:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:33.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:33.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:33.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:33.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:34.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:33.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:34.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:33.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:34.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:33.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:34.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:33.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:34.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:33.814+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:33.815919+0000) 2024-02-20T09:09:34.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:34.100+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:34.101591+0000) 2024-02-20T09:09:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:34.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:34.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:34.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:34.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:34.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:34.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:34.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:34.545+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:34.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:34.596+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:34.597254+0000) 2024-02-20T09:09:35.105 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:34.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:35.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:34.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:35.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:34.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:35.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:34.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:35.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:34.817+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:34.818405+0000) 2024-02-20T09:09:35.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:35.103+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.104376+0000) 2024-02-20T09:09:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:35.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:35.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:35.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:35.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:35.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:35.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:35.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:35.538+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:35.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:35.596+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.597511+0000) 2024-02-20T09:09:36.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:35.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:36.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:35.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:36.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:35.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:36.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:35.765+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:36.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:35.817+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.818799+0000) 2024-02-20T09:09:36.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:36.103+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.104793+0000) 2024-02-20T09:09:36.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:36.354+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.355840+0000) 2024-02-20T09:09:36.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:36.357+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 56970 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:36.356+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.356964+0000) 2024-02-20T09:09:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:36.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:36.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:36.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:36.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:36.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:36.356+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.357246+0000) 2024-02-20T09:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:36.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:36.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:36.509+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:36.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:36.509+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:36.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:36.596+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.597719+0000) 2024-02-20T09:09:37.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:36.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:37.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:36.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:37.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:36.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:37.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:36.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:37.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:36.818+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.818954+0000) 2024-02-20T09:09:37.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:37.104+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:37.105027+0000) 2024-02-20T09:09:37.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:37.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:37.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:37.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:37.714 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:37.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:37.714 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:37.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:37.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:37.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:37.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:37.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:37.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:37.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:37.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:37.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:37.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:37.599+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:37.600731+0000) 2024-02-20T09:09:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:37.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:37.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:37.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:37.712+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:37.821+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:37.822264+0000) 2024-02-20T09:09:38.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:38.107+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:38.108303+0000) 2024-02-20T09:09:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:38.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:38.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:38.444+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:38.444+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:38.603+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:38.604095+0000) 2024-02-20T09:09:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:38.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:38.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:38.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:38.360+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:39.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:38.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:39.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:38.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:39.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:38.720+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:39.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:38.721+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:39.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:38.824+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:38.825654+0000) 2024-02-20T09:09:39.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:39.110+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:39.111672+0000) 2024-02-20T09:09:39.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:39.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:39.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:39.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:39.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:39.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:39.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:39.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:39.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:39.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:39.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:39.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:39.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:39.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:39.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:39.475+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:39.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:39.605+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:39.606324+0000) 2024-02-20T09:09:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:39.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:39.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:39.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:39.711+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:39.828+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:39.829270+0000) 2024-02-20T09:09:40.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:40.114+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.115009+0000) 2024-02-20T09:09:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:40.460+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:40.460+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:40.460+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:40.461+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:40.605+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.606611+0000) 2024-02-20T09:09:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:40.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:40.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:40.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:40.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:40.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:41.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:40.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:41.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:40.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:41.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:40.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:41.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:40.751+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:41.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:40.828+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.829549+0000) 2024-02-20T09:09:41.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:41.114+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.115237+0000) 2024-02-20T09:09:41.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:41.361+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.362538+0000) 2024-02-20T09:09:41.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:41.362+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.362945+0000) 2024-02-20T09:09:41.695 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:41.365+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57009 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:41.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:41.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:41.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:41.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:41.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:41.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:41.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:41.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:41.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:41.364+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.365792+0000) 2024-02-20T09:09:41.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:41.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:41.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:41.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:41.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:41.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:41.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:41.475+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:41.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:41.605+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.606785+0000) 2024-02-20T09:09:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:41.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:41.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:41.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:41.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:41.828+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.829743+0000) 2024-02-20T09:09:42.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:42.114+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:42.115463+0000) 2024-02-20T09:09:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:42.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:42.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:42.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:42.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:42.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:42.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:42.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:42.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:42.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:42.608+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:42.609278+0000) 2024-02-20T09:09:43.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:42.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:43.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:42.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:43.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:42.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:43.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:42.726+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:43.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:42.832+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:42.833087+0000) 2024-02-20T09:09:43.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:43.116+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:43.117480+0000) 2024-02-20T09:09:43.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:43.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:43.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:43.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:43.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:43.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:43.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:43.450+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:43.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:43.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:43.692 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:43.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:43.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:43.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:43.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:43.460+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:43.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:43.611+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:43.612559+0000) 2024-02-20T09:09:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:43.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:43.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:43.688+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:43.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:43.835+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:43.836502+0000) 2024-02-20T09:09:44.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:44.118+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:44.119769+0000) 2024-02-20T09:09:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:44.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:44.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:44.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:44.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:44.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:44.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:44.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:44.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:44.615+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:44.615955+0000) 2024-02-20T09:09:45.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:44.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:45.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:44.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:45.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:44.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:45.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:44.732+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:45.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:44.838+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:44.839904+0000) 2024-02-20T09:09:45.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:45.121+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.122371+0000) 2024-02-20T09:09:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:45.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:45.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:45.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:45.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:45.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:45.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:45.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:45.470+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:45.615+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.616229+0000) 2024-02-20T09:09:46.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:45.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:46.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:45.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:46.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:45.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:46.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:45.778+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:46.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:45.839+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.840199+0000) 2024-02-20T09:09:46.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:46.369+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.370432+0000) 2024-02-20T09:09:46.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:46.121+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.122633+0000) 2024-02-20T09:09:46.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:46.370+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.371057+0000) 2024-02-20T09:09:46.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:46.371+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.372171+0000) 2024-02-20T09:09:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:46.372+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57049 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:46.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:46.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:46.501+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:46.501+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:46.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:46.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:46.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:46.480+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:46.615+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.616382+0000) 2024-02-20T09:09:47.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:46.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:47.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:46.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:47.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:46.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:47.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:46.747+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:47.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:46.839+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.840366+0000) 2024-02-20T09:09:47.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:47.121+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:47.122817+0000) 2024-02-20T09:09:47.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:47.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:47.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:47.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:47.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:47.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:47.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:47.506+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:47.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:47.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:47.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:47.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:47.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:47.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:47.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:47.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:47.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:47.618+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:47.619071+0000) 2024-02-20T09:09:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:47.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:47.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:47.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:47.709+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:47.841+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:47.842425+0000) 2024-02-20T09:09:48.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:48.124+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:48.125246+0000) 2024-02-20T09:09:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:48.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:48.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:48.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:48.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:48.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:48.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:48.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:48.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:48.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:48.621+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:48.622348+0000) 2024-02-20T09:09:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:48.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:48.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:48.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:48.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:48.845+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:48.845952+0000) 2024-02-20T09:09:49.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:49.126+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:49.127878+0000) 2024-02-20T09:09:49.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:49.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:49.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:49.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:49.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:49.459+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:49.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:49.460+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:49.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:49.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:49.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:49.490+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:49.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:49.625+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:49.625970+0000) 2024-02-20T09:09:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:49.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:49.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:49.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:49.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:49.848+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:49.849514+0000) 2024-02-20T09:09:50.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:50.129+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.130424+0000) 2024-02-20T09:09:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:50.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:50.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:50.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:50.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:50.467+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:50.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:50.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:50.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:50.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:50.659+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:50.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:50.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:50.506+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:50.507+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:50.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:50.625+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.626199+0000) 2024-02-20T09:09:51.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:50.849+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.849951+0000) 2024-02-20T09:09:51.381 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:51.377+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.378343+0000) 2024-02-20T09:09:51.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:51.129+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.130687+0000) 2024-02-20T09:09:51.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:51.376+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.377037+0000) 2024-02-20T09:09:51.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:51.378+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.379027+0000) 2024-02-20T09:09:51.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:51.379+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57088 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:51.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:51.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:51.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:51.452+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:51.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:51.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:51.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:51.621+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:51.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:51.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:51.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:51.514+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:51.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:51.625+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.626381+0000) 2024-02-20T09:09:52.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:51.849+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.850107+0000) 2024-02-20T09:09:52.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:52.129+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:52.130894+0000) 2024-02-20T09:09:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:52.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:52.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:52.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:52.419+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:52.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:52.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:52.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:52.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:52.625+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:52.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:52.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:52.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:52.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:52.627+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:52.628803+0000) 2024-02-20T09:09:53.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:52.852+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:52.852972+0000) 2024-02-20T09:09:53.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:53.133+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:53.134381+0000) 2024-02-20T09:09:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:53.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:53.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:53.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:53.619+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:53.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:53.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:53.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:53.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:53.446+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:53.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:53.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:53.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:53.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:53.630+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:53.630977+0000) 2024-02-20T09:09:54.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:53.854+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:53.855424+0000) 2024-02-20T09:09:54.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:54.136+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:54.137833+0000) 2024-02-20T09:09:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:54.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:54.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:54.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:54.451+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:54.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:54.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:54.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:54.638+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:54.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:54.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:54.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:54.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:54.633+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:54.634583+0000) 2024-02-20T09:09:55.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:54.857+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:54.858040+0000) 2024-02-20T09:09:55.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:55.140+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.141414+0000) 2024-02-20T09:09:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:55.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:55.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:55.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:55.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:55.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:55.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:55.628+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:55.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:55.628+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:55.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:55.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:55.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:55.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:55.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:55.633+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.634913+0000) 2024-02-20T09:09:56.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:55.857+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.858343+0000) 2024-02-20T09:09:56.399 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:09:56.388+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57128 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:09:56.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:56.140+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.141691+0000) 2024-02-20T09:09:56.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:56.383+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.384696+0000) 2024-02-20T09:09:56.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:56.383+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.384731+0000) 2024-02-20T09:09:56.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:56.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:56.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:56.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:56.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:56.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:56.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:56.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:56.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:56.383+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.384916+0000) 2024-02-20T09:09:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:56.634+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.635128+0000) 2024-02-20T09:09:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:56.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:56.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:56.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:56.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:56.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:56.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:56.593+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:56.594+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:57.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:56.857+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.858540+0000) 2024-02-20T09:09:57.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:57.140+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:57.141881+0000) 2024-02-20T09:09:57.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:57.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:57.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:57.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:57.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:57.390+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:57.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:57.391+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:57.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:57.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:57.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:57.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:57.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:57.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:57.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:57.554+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:57.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:57.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:57.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:57.636+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:57.637523+0000) 2024-02-20T09:09:58.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:57.859+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:57.860729+0000) 2024-02-20T09:09:58.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:58.143+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:58.144511+0000) 2024-02-20T09:09:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:58.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:58.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:58.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:58.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:58.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:58.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:58.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:58.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:58.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:58.592+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:58.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:58.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:58.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:58.414+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:58.639+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:58.639977+0000) 2024-02-20T09:09:59.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:58.861+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:58.862857+0000) 2024-02-20T09:09:59.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:59.146+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:59.147052+0000) 2024-02-20T09:09:59.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:59.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:59.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:59.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:59.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:59.381+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:09:59.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:59.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:09:59.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:59.643310+0000) 2024-02-20T09:09:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:59.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:59.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:59.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:09:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:09:59.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:09:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:59.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:59.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:59.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:09:59.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:59.621+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:00.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:09:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:09:59.865+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:59.866774+0000) 2024-02-20T09:10:00.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:00.148+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.149500+0000) 2024-02-20T09:10:00.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:00.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:00.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:00.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:00.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:00.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:00.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:00.385+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:00.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:00.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:00.508+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:00.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:00.509+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:00.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:00.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:00.654+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:00.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:00.654+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:00.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.643683+0000) 2024-02-20T09:10:01.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:00.866+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.867099+0000) 2024-02-20T09:10:01.404 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:01.395+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57167 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:01.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:01.392+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.393746+0000) 2024-02-20T09:10:01.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:01.392+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.393267+0000) 2024-02-20T09:10:01.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:01.148+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.149793+0000) 2024-02-20T09:10:01.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:01.392+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.393348+0000) 2024-02-20T09:10:01.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:01.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:01.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:01.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:01.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:01.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:01.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:01.466+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:01.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:01.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:01.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:01.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:01.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:01.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:01.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:01.434+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:01.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:01.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.643861+0000) 2024-02-20T09:10:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:01.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:01.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:01.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:01.667+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:01.866+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.867296+0000) 2024-02-20T09:10:02.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:02.149+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:02.150015+0000) 2024-02-20T09:10:02.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:02.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:02.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:02.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:02.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:02.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:02.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:02.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:02.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:02.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:02.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:02.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:02.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:02.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:02.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:02.388+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:02.645+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:02.646235+0000) 2024-02-20T09:10:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:02.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:02.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:02.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:02.662+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:02.868+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:02.869476+0000) 2024-02-20T09:10:03.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:03.151+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:03.152361+0000) 2024-02-20T09:10:03.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:03.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:03.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:03.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:03.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:03.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:03.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:03.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:03.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:03.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:03.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:03.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:03.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:03.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:03.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:03.647+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:03.648603+0000) 2024-02-20T09:10:03.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:03.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:03.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:03.678+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:03.679+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:03.870+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:03.871749+0000) 2024-02-20T09:10:04.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:04.153+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:04.154693+0000) 2024-02-20T09:10:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:04.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:04.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:04.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:04.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:04.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:04.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:04.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:04.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:04.640+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:04.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:04.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:04.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:04.423+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:04.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:04.650+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:04.651166+0000) 2024-02-20T09:10:05.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:04.874+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:04.875346+0000) 2024-02-20T09:10:05.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:05.155+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.156802+0000) 2024-02-20T09:10:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:05.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:05.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:05.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:05.592+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:05.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:05.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:05.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:05.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:05.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:05.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:05.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:05.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:05.444+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:05.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:05.650+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.651458+0000) 2024-02-20T09:10:06.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:05.874+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.875718+0000) 2024-02-20T09:10:06.411 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:06.402+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57207 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:06.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:06.399+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.400160+0000) 2024-02-20T09:10:06.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:06.156+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.157164+0000) 2024-02-20T09:10:06.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:06.399+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.400554+0000) 2024-02-20T09:10:06.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:06.399+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.400039+0000) 2024-02-20T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:06.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:06.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:06.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:06.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:06.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:06.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:06.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:06.463+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:06.650+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.651626+0000) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:06.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:06.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:06.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:06.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:06.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:06.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:07.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:06.874+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.875891+0000) 2024-02-20T09:10:07.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:07.156+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:07.157335+0000) 2024-02-20T09:10:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:07.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:07.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:07.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:07.419+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:07.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:07.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:07.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:07.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:07.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:07.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:07.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:07.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:07.653+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:07.653981+0000) 2024-02-20T09:10:08.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:07.877+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:07.878364+0000) 2024-02-20T09:10:08.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:08.159+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:08.159956+0000) 2024-02-20T09:10:08.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:08.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:08.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:08.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:08.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:08.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:08.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:08.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:08.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:08.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:08.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:08.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:08.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:08.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:08.485+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:08.486+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:08.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:08.655+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:08.656433+0000) 2024-02-20T09:10:09.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:08.880+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:08.881831+0000) 2024-02-20T09:10:09.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:09.161+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:09.162646+0000) 2024-02-20T09:10:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:09.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:09.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:09.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:09.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:09.423+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:09.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:09.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:09.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:09.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:09.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:09.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:09.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:09.467+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:09.658+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:09.659220+0000) 2024-02-20T09:10:10.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:09.883+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:09.884373+0000) 2024-02-20T09:10:10.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:10.164+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.165204+0000) 2024-02-20T09:10:10.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:10.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:10.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:10.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:10.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:10.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:10.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:10.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:10.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:10.576+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:10.577+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:10.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:10.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:10.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:10.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:10.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:10.500+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:10.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:10.658+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.659542+0000) 2024-02-20T09:10:11.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:10.883+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.884678+0000) 2024-02-20T09:10:11.419 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:11.409+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57246 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:11.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:11.407+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.408192+0000) 2024-02-20T09:10:11.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:11.164+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.165636+0000) 2024-02-20T09:10:11.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:11.407+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.408462+0000) 2024-02-20T09:10:11.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:11.407+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.408183+0000) 2024-02-20T09:10:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:11.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:11.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:11.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:11.627+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:11.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:11.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:11.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:11.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:11.450+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:11.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:11.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:11.515+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:11.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:11.515+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:11.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:11.658+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.659716+0000) 2024-02-20T09:10:12.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:11.883+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.884925+0000) 2024-02-20T09:10:12.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:12.164+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:12.165818+0000) 2024-02-20T09:10:12.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:12.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:12.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:12.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:12.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:12.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:12.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:12.606+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:12.607+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:12.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:12.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:12.489+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:12.490+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:12.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:12.660+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:12.661796+0000) 2024-02-20T09:10:13.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:12.886+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:12.887391+0000) 2024-02-20T09:10:13.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:13.168+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:13.169198+0000) 2024-02-20T09:10:13.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:13.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:13.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:13.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:13.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:13.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:13.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:13.391+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:13.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:13.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:13.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:13.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:13.662+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:13.663922+0000) 2024-02-20T09:10:13.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:13.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:13.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:13.649+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:13.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:13.649+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:14.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:13.890+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:13.890961+0000) 2024-02-20T09:10:14.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:14.171+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:14.172626+0000) 2024-02-20T09:10:14.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:14.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:14.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:14.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:14.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:14.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:14.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:14.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:14.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:14.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:14.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:14.496+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:14.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:14.665+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:14.665943+0000) 2024-02-20T09:10:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:14.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:14.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:14.695+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:14.695+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:14.892+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:14.893036+0000) 2024-02-20T09:10:15.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:15.173+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.174849+0000) 2024-02-20T09:10:15.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:15.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:15.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:15.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:15.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:15.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:15.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:15.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:15.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:15.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:15.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:15.458+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:15.665+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.666258+0000) 2024-02-20T09:10:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:15.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:15.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:15.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:15.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:15.892+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.893344+0000) 2024-02-20T09:10:16.426 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:16.416+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57287 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:16.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:16.414+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.415778+0000) 2024-02-20T09:10:16.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:16.174+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.175145+0000) 2024-02-20T09:10:16.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:16.414+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.415783+0000) 2024-02-20T09:10:16.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:16.414+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.415581+0000) 2024-02-20T09:10:16.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:16.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:16.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:16.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:16.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:16.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:16.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:16.471+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:16.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:16.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:16.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:16.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:16.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:16.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:16.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:16.498+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:16.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:16.665+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.666428+0000) 2024-02-20T09:10:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:16.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:16.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:16.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:16.713+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:16.892+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.893539+0000) 2024-02-20T09:10:17.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:17.174+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:17.175347+0000) 2024-02-20T09:10:17.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:17.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:17.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:17.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:17.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:17.466+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:17.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:17.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:17.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:17.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:17.484+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:17.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:17.667+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:17.668838+0000) 2024-02-20T09:10:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:17.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:17.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:17.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:17.741+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:17.895+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:17.895968+0000) 2024-02-20T09:10:18.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:18.176+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:18.177556+0000) 2024-02-20T09:10:18.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:18.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:18.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:18.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:18.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:18.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:18.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:18.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:18.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:18.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:18.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:18.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:18.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:18.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:18.670+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:18.671509+0000) 2024-02-20T09:10:18.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:18.701+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:18.701+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:18.701+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:18.702+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:18.897+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:18.897970+0000) 2024-02-20T09:10:19.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:19.179+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:19.180448+0000) 2024-02-20T09:10:19.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:19.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:19.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:19.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:19.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:19.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:19.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:19.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:19.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:19.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:19.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:19.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:19.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:19.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:19.672+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:19.673844+0000) 2024-02-20T09:10:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:19.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:19.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:19.717+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:19.718+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:19.899+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:19.900223+0000) 2024-02-20T09:10:20.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:20.182+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.183021+0000) 2024-02-20T09:10:20.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:20.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:20.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:20.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:20.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:20.554+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:20.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:20.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:20.490+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:20.491+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:20.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:20.673+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.674282+0000) 2024-02-20T09:10:21.184 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:20.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:21.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:20.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:21.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:20.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:21.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:20.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:21.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:20.899+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.900526+0000) 2024-02-20T09:10:21.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:21.423+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57326 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:21.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:21.420+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.421378+0000) 2024-02-20T09:10:21.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:21.182+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.183430+0000) 2024-02-20T09:10:21.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:21.420+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.421609+0000) 2024-02-20T09:10:21.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:21.420+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.421822+0000) 2024-02-20T09:10:21.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:21.571+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:21.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:21.571+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:21.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:21.571+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:21.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:21.572+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:21.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:21.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:21.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:21.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:21.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:21.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:21.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:21.475+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:21.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:21.673+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.674499+0000) 2024-02-20T09:10:21.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:21.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:21.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:21.716+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:21.717+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:21.899+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.900732+0000) 2024-02-20T09:10:22.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:22.182+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:22.183617+0000) 2024-02-20T09:10:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:22.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:22.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:22.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:22.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:22.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:22.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:22.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:22.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:22.449+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:22.450+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:22.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:22.676+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:22.676989+0000) 2024-02-20T09:10:23.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:22.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:23.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:22.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:23.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:22.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:23.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:22.769+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:23.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:22.901+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:22.903184+0000) 2024-02-20T09:10:23.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:23.184+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:23.186135+0000) 2024-02-20T09:10:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:23.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:23.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:23.643+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:23.644+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:23.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:23.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:23.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:23.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:23.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:23.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:23.466+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:23.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:23.677+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:23.679615+0000) 2024-02-20T09:10:24.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:23.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:24.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:23.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:24.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:23.797+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:24.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:23.798+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:24.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:23.904+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:23.905663+0000) 2024-02-20T09:10:24.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:24.186+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:24.188686+0000) 2024-02-20T09:10:24.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:24.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:24.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:24.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:24.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:24.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:24.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:24.674+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:24.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:24.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:24.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:24.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:24.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:24.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:24.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:24.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:24.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:24.680+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:24.682174+0000) 2024-02-20T09:10:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:24.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:24.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:24.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:24.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:24.907+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:24.909327+0000) 2024-02-20T09:10:25.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:25.190+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.191218+0000) 2024-02-20T09:10:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:25.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:25.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:25.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:25.660+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:25.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:25.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:25.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:25.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:25.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:25.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:25.680+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.682626+0000) 2024-02-20T09:10:26.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:25.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:26.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:25.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:26.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:25.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:26.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:25.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:26.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:25.908+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.909750+0000) 2024-02-20T09:10:26.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:26.189+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.191565+0000) 2024-02-20T09:10:26.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:26.428+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.429259+0000) 2024-02-20T09:10:26.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:26.429+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.430389+0000) 2024-02-20T09:10:26.451 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:26.430+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57366 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:26.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:26.428+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.429208+0000) 2024-02-20T09:10:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:26.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:26.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:26.627+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:26.628+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:26.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:26.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:26.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:26.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:26.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:26.681+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.682866+0000) 2024-02-20T09:10:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:26.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:26.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:26.811+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:26.811+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:26.909+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.910018+0000) 2024-02-20T09:10:27.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:27.190+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:27.191760+0000) 2024-02-20T09:10:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:27.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:27.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:27.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:27.651+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:27.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:27.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:27.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:27.509+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:27.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:27.685+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:27.686335+0000) 2024-02-20T09:10:28.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:27.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:28.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:27.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:28.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:27.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:28.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:27.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:28.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:27.912+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:27.913537+0000) 2024-02-20T09:10:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:28.194+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:28.195323+0000) 2024-02-20T09:10:28.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:28.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:28.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:28.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:28.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:28.677+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:28.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:28.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:28.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:28.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:28.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:28.688+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:28.689593+0000) 2024-02-20T09:10:29.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:28.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:29.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:28.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:29.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:28.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:29.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:28.895+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:29.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:28.915+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:28.916092+0000) 2024-02-20T09:10:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:29.197+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:29.197955+0000) 2024-02-20T09:10:29.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:29.703+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:29.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:29.703+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:29.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:29.703+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:29.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:29.704+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:29.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:29.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:29.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:29.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:29.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:29.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:29.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:29.471+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:29.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:29.690+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:29.691637+0000) 2024-02-20T09:10:30.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:29.917+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:29.918211+0000) 2024-02-20T09:10:30.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:29.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:30.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:29.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:30.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:29.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:30.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:29.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:30.199+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.200454+0000) 2024-02-20T09:10:30.750 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:30.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:30.750 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:30.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:30.750 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:30.487+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:30.750 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:30.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:30.750 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:30.691+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.692062+0000) 2024-02-20T09:10:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:30.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:30.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:30.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:30.749+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:31.199+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.200835+0000) 2024-02-20T09:10:31.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:30.917+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.918667+0000) 2024-02-20T09:10:31.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:30.939+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:31.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:30.939+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:31.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:30.939+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:31.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:30.940+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:31.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:31.434+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.435474+0000) 2024-02-20T09:10:31.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:31.435+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.436536+0000) 2024-02-20T09:10:31.461 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:31.438+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57405 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:31.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:31.435+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.436211+0000) 2024-02-20T09:10:31.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:31.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:31.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:31.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:31.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:31.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:31.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:31.439+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:31.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:31.691+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.692213+0000) 2024-02-20T09:10:32.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:31.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:32.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:31.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:32.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:31.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:32.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:31.771+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:32.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:31.917+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.918901+0000) 2024-02-20T09:10:32.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:31.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:32.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:31.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:32.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:31.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:32.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:31.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:32.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:32.200+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:32.201024+0000) 2024-02-20T09:10:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:32.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:32.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:32.457+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:32.457+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:32.693+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:32.694583+0000) 2024-02-20T09:10:32.983 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:32.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:32.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:32.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:32.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:32.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:32.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:32.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:32.984 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:32.920+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:32.921681+0000) 2024-02-20T09:10:33.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:33.202+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:33.203500+0000) 2024-02-20T09:10:33.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:32.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:33.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:32.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:33.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:32.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:33.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:32.982+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:33.765 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:33.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:33.765 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:33.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:33.765 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:33.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:33.765 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:33.503+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:33.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:33.696+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:33.697107+0000) 2024-02-20T09:10:34.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:33.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:34.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:33.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:34.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:33.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:34.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:33.763+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:34.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:33.924+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:33.925364+0000) 2024-02-20T09:10:34.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:33.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:34.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:33.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:34.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:33.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:34.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:33.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:34.205+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:34.206150+0000) 2024-02-20T09:10:34.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:34.727+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:34.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:34.727+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:34.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:34.727+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:34.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:34.727+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:34.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:34.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:34.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:34.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:34.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:34.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:34.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:34.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:34.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:34.698+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:34.699596+0000) 2024-02-20T09:10:35.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:34.927+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:34.928009+0000) 2024-02-20T09:10:35.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:34.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:35.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:34.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:35.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:34.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:35.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:34.954+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:35.207+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.208687+0000) 2024-02-20T09:10:35.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:35.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:35.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:35.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:35.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:35.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:35.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:35.737+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:35.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:35.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:35.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:35.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:35.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:35.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:35.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:35.598+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:35.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:35.698+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.699933+0000) 2024-02-20T09:10:36.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:35.927+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.928544+0000) 2024-02-20T09:10:36.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:35.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:36.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:35.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:36.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:35.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:36.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:35.973+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:36.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:36.208+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.209134+0000) 2024-02-20T09:10:36.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:36.442+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.443694+0000) 2024-02-20T09:10:36.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:36.442+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.443017+0000) 2024-02-20T09:10:36.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:36.446+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57445 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:36.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:36.443+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.444805+0000) 2024-02-20T09:10:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:36.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:36.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:36.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:36.558+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:36.699+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.700117+0000) 2024-02-20T09:10:37.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:36.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:37.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:36.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:37.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:36.728+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:37.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:36.728+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:37.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:36.927+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.928788+0000) 2024-02-20T09:10:37.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:36.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:37.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:36.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:37.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:36.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:37.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:36.967+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:37.208+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:37.209323+0000) 2024-02-20T09:10:37.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:37.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:37.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:37.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:37.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:37.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:37.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:37.570+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:37.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:37.701+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:37.702459+0000) 2024-02-20T09:10:38.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:37.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:38.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:37.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:38.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:37.771+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:38.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:37.772+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:38.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:37.930+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:37.931242+0000) 2024-02-20T09:10:38.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:37.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:38.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:37.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:38.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:37.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:38.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:37.945+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:38.210+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:38.211828+0000) 2024-02-20T09:10:38.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:38.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:38.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:38.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:38.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:38.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:38.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:38.785+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:38.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:38.598+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:38.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:38.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:38.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:38.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:38.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:38.600+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:38.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:38.704+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:38.704928+0000) 2024-02-20T09:10:39.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:38.932+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:38.933660+0000) 2024-02-20T09:10:39.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:38.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:39.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:38.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:39.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:38.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:39.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:38.960+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:39.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:39.213+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:39.214545+0000) 2024-02-20T09:10:39.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:39.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:39.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:39.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:39.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:39.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:39.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:39.553+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:39.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:39.706+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:39.707394+0000) 2024-02-20T09:10:40.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:39.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:40.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:39.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:40.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:39.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:40.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:39.819+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:40.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:39.935+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:39.936159+0000) 2024-02-20T09:10:40.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:39.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:40.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:39.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:40.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:39.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:40.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:39.979+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:40.216+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.217087+0000) 2024-02-20T09:10:40.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:40.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:40.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:40.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:40.822 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:40.796+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:40.822 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:40.797+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:40.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:40.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:40.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:40.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:40.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:40.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:40.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:40.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:40.823 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:40.706+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.707842+0000) 2024-02-20T09:10:41.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:40.935+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.936457+0000) 2024-02-20T09:10:41.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:41.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:41.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:41.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:41.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:41.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:41.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:41.008+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:41.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:41.453+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57484 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:41.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:41.451+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.452212+0000) 2024-02-20T09:10:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:41.451+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.452549+0000) 2024-02-20T09:10:41.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:41.216+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.217536+0000) 2024-02-20T09:10:41.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:41.451+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.452170+0000) 2024-02-20T09:10:41.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:41.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:41.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:41.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:41.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:41.517+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:41.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:41.517+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:41.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:41.707+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.707968+0000) 2024-02-20T09:10:42.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:41.935+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.936645+0000) 2024-02-20T09:10:42.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:42.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:42.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:42.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:42.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:42.039+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:41.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:41.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:42.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:41.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:42.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:41.771+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:42.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:42.217+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:42.218268+0000) 2024-02-20T09:10:42.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:42.040+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:42.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:42.554+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:42.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:42.554+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:42.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:42.554+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:42.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:42.555+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:42.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:42.938+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:42.939255+0000) 2024-02-20T09:10:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:42.710+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:42.711337+0000) 2024-02-20T09:10:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:42.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:42.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:42.793+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:42.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:42.794+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:43.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:43.219+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:43.220351+0000) 2024-02-20T09:10:43.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:43.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:43.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:43.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:43.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:43.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:43.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:43.088+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:43.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:43.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:43.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:43.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:43.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:43.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:43.716 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:43.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:43.941+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:43.942013+0000) 2024-02-20T09:10:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:43.712+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:43.713860+0000) 2024-02-20T09:10:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:43.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:43.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:43.821+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:43.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:43.821+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:44.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:44.222+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:44.222968+0000) 2024-02-20T09:10:44.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:44.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:44.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:44.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:44.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:44.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:44.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:44.072+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:44.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:44.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:44.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:44.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:44.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:44.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:44.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:44.533+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:44.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:44.716+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:44.717197+0000) 2024-02-20T09:10:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:44.943+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:44.944369+0000) 2024-02-20T09:10:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:44.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:44.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:44.782+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:44.783+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:45.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:45.224+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.225730+0000) 2024-02-20T09:10:45.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:45.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:45.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:45.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:45.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:45.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:45.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:45.045+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:45.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:45.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:45.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:45.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:45.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:45.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:45.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:45.484+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:45.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:45.716+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.717633+0000) 2024-02-20T09:10:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:45.943+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.944746+0000) 2024-02-20T09:10:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:45.801+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:45.801+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:45.801+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:45.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:45.802+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:46.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:46.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.226117+0000) 2024-02-20T09:10:46.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:46.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:46.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:46.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:46.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:46.095+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:46.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:46.096+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:46.719 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:46.461+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57524 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:46.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:46.457+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.458332+0000) 2024-02-20T09:10:46.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:46.458+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.459825+0000) 2024-02-20T09:10:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:46.458+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.459018+0000) 2024-02-20T09:10:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:46.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:46.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:46.479+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:46.480+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:46.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:46.716+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.717843+0000) 2024-02-20T09:10:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:46.944+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.944966+0000) 2024-02-20T09:10:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:46.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:46.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:46.819+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:46.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:47.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:47.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:47.226311+0000) 2024-02-20T09:10:47.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:47.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:47.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:47.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:47.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:47.078+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:47.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:47.079+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:47.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:47.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:47.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:47.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:48.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:47.719+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:47.720877+0000) 2024-02-20T09:10:48.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:47.946+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:47.947032+0000) 2024-02-20T09:10:48.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:48.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:48.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:48.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:48.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:48.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:48.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:47.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:48.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:47.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:48.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:47.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:48.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:47.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:48.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:48.227+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:48.228890+0000) 2024-02-20T09:10:48.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:48.111+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:48.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:48.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:48.505+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:48.506+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:49.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:48.723+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:48.724247+0000) 2024-02-20T09:10:49.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:48.948+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:48.949414+0000) 2024-02-20T09:10:49.069 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:49.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:49.069 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:49.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:49.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:48.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:49.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:48.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:49.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:48.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:49.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:48.812+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:49.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:49.230+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:49.231114+0000) 2024-02-20T09:10:49.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:49.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:49.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:49.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:49.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:49.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:49.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:49.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:50.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:49.725+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:49.726888+0000) 2024-02-20T09:10:50.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:49.951+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:49.951965+0000) 2024-02-20T09:10:50.069 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:50.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:50.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:49.836+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:50.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:49.836+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:50.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:49.836+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:50.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:49.836+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:50.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:50.232+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.233452+0000) 2024-02-20T09:10:50.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:50.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:50.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:50.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:50.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:50.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:50.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:50.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:50.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:50.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:50.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:51.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:50.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:51.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:50.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:51.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:50.883+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:51.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:50.884+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:51.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:50.951+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.952431+0000) 2024-02-20T09:10:51.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:50.726+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.727284+0000) 2024-02-20T09:10:51.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:51.232+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.233824+0000) 2024-02-20T09:10:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:51.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:51.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:51.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:51.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:51.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:51.468+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57563 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:51.465+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.466804+0000) 2024-02-20T09:10:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:51.465+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.466760+0000) 2024-02-20T09:10:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:51.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:51.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:51.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:51.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:51.466+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.466967+0000) 2024-02-20T09:10:52.090 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:51.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:52.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:51.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:52.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:51.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:52.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:51.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:52.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:51.726+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.727470+0000) 2024-02-20T09:10:52.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:51.951+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.952652+0000) 2024-02-20T09:10:52.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:52.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:52.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:52.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:52.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:52.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:52.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:52.233+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:52.233969+0000) 2024-02-20T09:10:52.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:52.089+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:52.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:52.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:52.427+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:52.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:52.428+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:53.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:52.955+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:52.956060+0000) 2024-02-20T09:10:53.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:52.729+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:52.730572+0000) 2024-02-20T09:10:53.090 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:52.875+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:53.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:52.875+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:53.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:52.875+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:53.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:52.875+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:53.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:53.236+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:53.237481+0000) 2024-02-20T09:10:53.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:53.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:53.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:53.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:53.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:53.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:53.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:53.089+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:53.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:53.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:53.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:53.462+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:54.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:53.731+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:53.732665+0000) 2024-02-20T09:10:54.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:53.957+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:53.958509+0000) 2024-02-20T09:10:54.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:54.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:54.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:54.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:54.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:53.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:54.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:53.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:54.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:53.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:54.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:53.902+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:54.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:54.239+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:54.240786+0000) 2024-02-20T09:10:54.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:54.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:54.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:54.080+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:54.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:54.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:54.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:54.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:54.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:54.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:54.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:54.938+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:55.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:54.959+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:54.960528+0000) 2024-02-20T09:10:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:55.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:55.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:54.733+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:54.734699+0000) 2024-02-20T09:10:55.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:55.241+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.242434+0000) 2024-02-20T09:10:55.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:55.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:55.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:55.039+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:55.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:55.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:55.421+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:55.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:56.007 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:55.734+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.735065+0000) 2024-02-20T09:10:56.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:55.959+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.960924+0000) 2024-02-20T09:10:56.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:56.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:56.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:56.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:56.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:55.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:56.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:55.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:56.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:55.943+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:56.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:55.944+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:56.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:56.242+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.242972+0000) 2024-02-20T09:10:56.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:56.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:56.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:56.006+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:56.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:10:56.476+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57603 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:10:56.600 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:56.474+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.475087+0000) 2024-02-20T09:10:56.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:56.474+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.475884+0000) 2024-02-20T09:10:56.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:56.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:56.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:56.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:56.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:56.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:56.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:56.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:56.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:56.473+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.474371+0000) 2024-02-20T09:10:56.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:56.734+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.735238+0000) 2024-02-20T09:10:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:56.914+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:56.914+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:56.914+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:56.915+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:56.960+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.961096+0000) 2024-02-20T09:10:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:57.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:57.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:57.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:57.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:57.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:57.242+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:57.243152+0000) 2024-02-20T09:10:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:57.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:57.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:57.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:57.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:57.415+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:57.823 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:57.736+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:57.737464+0000) 2024-02-20T09:10:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:57.963+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:57.964574+0000) 2024-02-20T09:10:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:58.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:58.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:58.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:58.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:58.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:58.046+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:58.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:57.891+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:58.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:57.891+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:58.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:57.891+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:58.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:57.892+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:58.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:58.245+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:58.246405+0000) 2024-02-20T09:10:58.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:58.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:58.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:58.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:58.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:58.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:58.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:58.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:58.847 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:58.740+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:58.741028+0000) 2024-02-20T09:10:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:58.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:58.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:58.845+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:58.846+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:58.965+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:58.966606+0000) 2024-02-20T09:10:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:59.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:59.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:59.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:10:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:59.094+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:59.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:59.247+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:59.248427+0000) 2024-02-20T09:10:59.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:59.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:59.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:59.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:59.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:59.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:10:59.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:59.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:10:59.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:10:59.743+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:59.744463+0000) 2024-02-20T09:10:59.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:59.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:59.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:59.800+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:10:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:10:59.801+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:00.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:10:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:10:59.968+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:59.969193+0000) 2024-02-20T09:11:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:00.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:00.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:00.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:00.049+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:00.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:00.249+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.250814+0000) 2024-02-20T09:11:00.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:00.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:00.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:00.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:00.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:00.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:00.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:00.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:00.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:00.744+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.744968+0000) 2024-02-20T09:11:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:00.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:00.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:00.772+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:00.773+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:00.968+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.969669+0000) 2024-02-20T09:11:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:01.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:01.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:01.049+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:01.050+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:01.503 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:01.484+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57642 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:01.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:01.481+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.482178+0000) 2024-02-20T09:11:01.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:01.250+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.251227+0000) 2024-02-20T09:11:01.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:01.481+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.482443+0000) 2024-02-20T09:11:01.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:01.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:01.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:01.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:01.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:01.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:01.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:01.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:01.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:01.480+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.481787+0000) 2024-02-20T09:11:01.823 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:01.744+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.745156+0000) 2024-02-20T09:11:01.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:01.727+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:01.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:01.727+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:01.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:01.727+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:01.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:01.728+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:01.968+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.969885+0000) 2024-02-20T09:11:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:02.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:02.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:02.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:02.031+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:02.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:02.250+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:02.251412+0000) 2024-02-20T09:11:02.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:02.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:02.579 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:02.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:02.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:02.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:02.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:02.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:02.746+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:02.747166+0000) 2024-02-20T09:11:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:02.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:02.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:02.737+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:02.738+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:03.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:02.970+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:02.971880+0000) 2024-02-20T09:11:03.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:03.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:03.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:03.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.025+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:03.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:03.253+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:03.254052+0000) 2024-02-20T09:11:03.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:03.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:03.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:03.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:03.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:03.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:03.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:03.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:03.748+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:03.749347+0000) 2024-02-20T09:11:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:03.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:03.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:03.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:03.719+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:04.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.974+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:03.975367+0000) 2024-02-20T09:11:04.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:04.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:04.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:04.252 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:03.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:04.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:04.255+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:04.256140+0000) 2024-02-20T09:11:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:04.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:04.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:04.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:04.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:04.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:04.752+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:04.752970+0000) 2024-02-20T09:11:04.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:04.755+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:04.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:04.755+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:04.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:04.755+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:04.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:04.756+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:05.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:04.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:04.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:04.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:04.947+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:04.976+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:04.977425+0000) 2024-02-20T09:11:05.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:05.257+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.258349+0000) 2024-02-20T09:11:05.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:05.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:05.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:05.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:05.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:05.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:05.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:05.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:05.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:05.752+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.753349+0000) 2024-02-20T09:11:05.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:05.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:05.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:05.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:05.937 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:05.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:05.937 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:05.734+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:05.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:05.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:05.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:05.935+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:05.977+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.977949+0000) 2024-02-20T09:11:06.513 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:06.492+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57682 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:06.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:06.488+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.489210+0000) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:06.257+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.258900+0000) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:06.488+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.489363+0000) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:06.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:06.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:06.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:06.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:06.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:06.489+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.490125+0000) 2024-02-20T09:11:06.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:06.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:06.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:06.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:06.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:06.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:06.825 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:06.744+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:06.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:06.752+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.753550+0000) 2024-02-20T09:11:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:06.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:06.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:06.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:06.911+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:06.977+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.978119+0000) 2024-02-20T09:11:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:07.258+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:07.259074+0000) 2024-02-20T09:11:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:07.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:07.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:07.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:07.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:07.227+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:07.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:07.754+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:07.755693+0000) 2024-02-20T09:11:08.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:07.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:08.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:07.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:08.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:07.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:08.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:07.770+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:08.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:07.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:08.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:07.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:08.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:07.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:07.959+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:07.980+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:07.981431+0000) 2024-02-20T09:11:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:08.260+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:08.261086+0000) 2024-02-20T09:11:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:08.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:08.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:08.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:08.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:08.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:08.757+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:08.757958+0000) 2024-02-20T09:11:08.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:08.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:08.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:08.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:08.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:08.778+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:08.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:08.779+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:09.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:08.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:09.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:08.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:09.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:08.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:09.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:08.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:09.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:08.983+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:08.984873+0000) 2024-02-20T09:11:09.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:09.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:09.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:09.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:09.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:09.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:09.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:09.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:09.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:09.262+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:09.263536+0000) 2024-02-20T09:11:09.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:09.759+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:09.760115+0000) 2024-02-20T09:11:09.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:09.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:09.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:09.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:09.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:09.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:09.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:09.784+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:10.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:09.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:10.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:09.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:10.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:09.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:10.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:09.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:10.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:09.986+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:09.987114+0000) 2024-02-20T09:11:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:10.266+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.267107+0000) 2024-02-20T09:11:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:10.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:10.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:10.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:10.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:10.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:10.759+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.760523+0000) 2024-02-20T09:11:10.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:10.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:10.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:10.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:10.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:10.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:10.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:10.775+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:10.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:10.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:10.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:10.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:11.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:10.986+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.987674+0000) 2024-02-20T09:11:11.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:11.266+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.267673+0000) 2024-02-20T09:11:11.507 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:11.497+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.498442+0000) 2024-02-20T09:11:11.507 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:11.499+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57721 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:11.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:11.497+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.498568+0000) 2024-02-20T09:11:11.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:11.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:11.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:11.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:11.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:11.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:11.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:11.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:11.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:11.496+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.497777+0000) 2024-02-20T09:11:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:11.759+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.760681+0000) 2024-02-20T09:11:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:11.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:11.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:11.786+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:11.787+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:11.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:12.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:11.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:12.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:11.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:12.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:11.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:12.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:11.986+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.987862+0000) 2024-02-20T09:11:12.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:12.266+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:12.267863+0000) 2024-02-20T09:11:12.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:12.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:12.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:12.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:12.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:12.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:12.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:12.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:12.866 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:12.762+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:12.763902+0000) 2024-02-20T09:11:12.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:12.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:12.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:12.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:12.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:12.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:12.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:12.803+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:12.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:12.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:12.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:12.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:12.989+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:12.990265+0000) 2024-02-20T09:11:13.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:13.269+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:13.270244+0000) 2024-02-20T09:11:13.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:13.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:13.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:13.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:13.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:13.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:13.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:13.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:13.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:13.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:13.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:13.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:13.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:13.805+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:13.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:13.805+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:13.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:13.765+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:13.765975+0000) 2024-02-20T09:11:14.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:13.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:14.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:13.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:14.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:13.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:14.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:13.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:14.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:13.991+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:13.992350+0000) 2024-02-20T09:11:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:14.272+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:14.273821+0000) 2024-02-20T09:11:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:14.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:14.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:14.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:14.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:14.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:14.767+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:14.767954+0000) 2024-02-20T09:11:14.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:14.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:14.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:14.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:14.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:14.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:14.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:14.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:14.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:14.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:14.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:14.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:14.994+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:14.995891+0000) 2024-02-20T09:11:15.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:15.275+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.276155+0000) 2024-02-20T09:11:15.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:15.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:15.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:15.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:15.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:15.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:15.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:15.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:15.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:15.767+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.768349+0000) 2024-02-20T09:11:16.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:15.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:15.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:15.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:16.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:15.840+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:15.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:15.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:15.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:15.957+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:16.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:15.994+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.996513+0000) 2024-02-20T09:11:16.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:16.274+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.276570+0000) 2024-02-20T09:11:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:16.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:16.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:16.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:16.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:16.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:16.507+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57761 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:16.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:16.503+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.505376+0000) 2024-02-20T09:11:16.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:16.502+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.504864+0000) 2024-02-20T09:11:16.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:16.502+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.504826+0000) 2024-02-20T09:11:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:16.766+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.768535+0000) 2024-02-20T09:11:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:16.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:16.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:16.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:16.866+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:16.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:16.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:16.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:16.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:17.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:16.994+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.996685+0000) 2024-02-20T09:11:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:17.274+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:17.276759+0000) 2024-02-20T09:11:17.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:17.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:17.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:17.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:17.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:17.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:17.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:18.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:17.769+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:17.771043+0000) 2024-02-20T09:11:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:17.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:17.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:17.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:17.857+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:17.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:17.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:17.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:17.952+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:18.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:17.998+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:17.999998+0000) 2024-02-20T09:11:18.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:18.277+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:18.279335+0000) 2024-02-20T09:11:18.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:18.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:18.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:18.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:18.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:18.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:18.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:18.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:18.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:18.771+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:18.773394+0000) 2024-02-20T09:11:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:18.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:18.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:18.862+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:18.863+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:18.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:18.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:18.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:18.916+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:19.001+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:19.003386+0000) 2024-02-20T09:11:19.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:19.280+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:19.281969+0000) 2024-02-20T09:11:19.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:19.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:19.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:19.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:19.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:19.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:19.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:19.298+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:19.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:19.774+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:19.775611+0000) 2024-02-20T09:11:20.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:19.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:19.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:19.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:19.842+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:19.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:19.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:19.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:19.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:20.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:20.005+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.006815+0000) 2024-02-20T09:11:20.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:20.283+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.284462+0000) 2024-02-20T09:11:20.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:20.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:20.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:20.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:20.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:20.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:20.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:20.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:20.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:20.813+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:20.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:20.813+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:20.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:20.813+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:20.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:20.813+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:20.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:20.775+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.776135+0000) 2024-02-20T09:11:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:20.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:20.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:20.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:20.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:21.006+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.007401+0000) 2024-02-20T09:11:21.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:21.284+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.285032+0000) 2024-02-20T09:11:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:21.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:21.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:21.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:21.227+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:21.777 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:21.516+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57800 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:21.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:21.513+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.514301+0000) 2024-02-20T09:11:21.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:21.513+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.513977+0000) 2024-02-20T09:11:21.778 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:21.513+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.514536+0000) 2024-02-20T09:11:22.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:21.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:21.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:21.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:22.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:21.798+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:21.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:21.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:21.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:21.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:22.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:22.006+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:22.007599+0000) 2024-02-20T09:11:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:21.775+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.776321+0000) 2024-02-20T09:11:22.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:22.284+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:22.285217+0000) 2024-02-20T09:11:22.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:22.264+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:22.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:22.264+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:22.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:22.264+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:22.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:22.265+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:22.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:22.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:22.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:22.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:22.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:22.806+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:22.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:22.807+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:22.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:22.778+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:22.779745+0000) 2024-02-20T09:11:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:22.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:22.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:22.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:22.958+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:23.008+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:23.009861+0000) 2024-02-20T09:11:23.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:23.286+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:23.287594+0000) 2024-02-20T09:11:23.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:23.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:23.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:23.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:23.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:23.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:23.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:23.250+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:23.781+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:23.782254+0000) 2024-02-20T09:11:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:23.822+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:23.822+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:23.822+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:23.823+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:24.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:23.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:24.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:23.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:24.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:23.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:24.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:23.982+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:24.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:24.011+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:24.012030+0000) 2024-02-20T09:11:24.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:24.290+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:24.290961+0000) 2024-02-20T09:11:24.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:24.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:24.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:24.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:24.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:24.296+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:24.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:24.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:24.784+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:24.784932+0000) 2024-02-20T09:11:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:24.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:24.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:24.811+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:24.812+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:25.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:24.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:25.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:24.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:25.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:24.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:25.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:24.982+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:25.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:25.014+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.015610+0000) 2024-02-20T09:11:25.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:25.292+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.293227+0000) 2024-02-20T09:11:25.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:25.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:25.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:25.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:25.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:25.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:25.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:25.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:25.784+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.785583+0000) 2024-02-20T09:11:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:25.809+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:25.809+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:25.809+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:25.810+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:26.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:25.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:26.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:25.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:26.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:25.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:26.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:25.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:26.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:26.015+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.016046+0000) 2024-02-20T09:11:26.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:26.292+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.293867+0000) 2024-02-20T09:11:26.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:26.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:26.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:26.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:26.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:26.320+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:26.520 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:26.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:26.787 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:26.522+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57840 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:26.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:26.520+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.521655+0000) 2024-02-20T09:11:26.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:26.538+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.539929+0000) 2024-02-20T09:11:26.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:26.520+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.521483+0000) 2024-02-20T09:11:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:26.784+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.785752+0000) 2024-02-20T09:11:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:26.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:26.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:26.839+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:26.840+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:26.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:27.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:26.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:26.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:26.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:27.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:27.015+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:27.016249+0000) 2024-02-20T09:11:27.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:27.293+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:27.294070+0000) 2024-02-20T09:11:27.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:27.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:27.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:27.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:27.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:27.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:27.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:28.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:27.788+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:27.789177+0000) 2024-02-20T09:11:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:27.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:27.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:27.818+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:27.820+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:27.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:27.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:27.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:27.990+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:28.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:28.017+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:28.018196+0000) 2024-02-20T09:11:28.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:28.295+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:28.296669+0000) 2024-02-20T09:11:28.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:28.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:28.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:28.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:28.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:28.407+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:28.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:28.408+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:28.837 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:28.791+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:28.792070+0000) 2024-02-20T09:11:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:28.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:28.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:28.835+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:28.836+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:29.020+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:29.021544+0000) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:29.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:29.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:29.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:29.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:29.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:29.298+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:29.299402+0000) 2024-02-20T09:11:29.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:29.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:29.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:29.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:29.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:29.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:29.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:29.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:29.794+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:29.795458+0000) 2024-02-20T09:11:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:29.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:29.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:29.827+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:29.828+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:30.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:30.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:30.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:30.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:30.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:30.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:30.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:30.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:30.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:30.024+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.025312+0000) 2024-02-20T09:11:30.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:30.301+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.302110+0000) 2024-02-20T09:11:30.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:30.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:30.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:30.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:30.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:30.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:30.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:30.361+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:30.863 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:30.795+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.796013+0000) 2024-02-20T09:11:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:30.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:31.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:30.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:30.861+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:30.862+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:31.025+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.025971+0000) 2024-02-20T09:11:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:31.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:31.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:31.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:31.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:31.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:31.059+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:31.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:31.301+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.302670+0000) 2024-02-20T09:11:31.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:31.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:31.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:31.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:31.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:31.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:31.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:31.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:31.797 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:31.530+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57880 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:31.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:31.528+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.529323+0000) 2024-02-20T09:11:31.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:31.527+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.528290+0000) 2024-02-20T09:11:31.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:31.527+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.528520+0000) 2024-02-20T09:11:32.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:32.025+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:32.026084+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:32.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:32.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:32.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:31.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:31.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:31.834+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:32.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:31.835+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:32.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:31.795+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.796205+0000) 2024-02-20T09:11:32.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:32.301+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:32.302862+0000) 2024-02-20T09:11:32.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:32.089+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:32.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:32.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:32.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:32.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:32.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:32.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:32.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:32.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:33.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:33.028+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:33.029037+0000) 2024-02-20T09:11:33.055 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:32.797+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:32.798716+0000) 2024-02-20T09:11:33.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:32.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:33.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:32.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:33.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:32.846+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:33.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:32.847+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:33.306 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:33.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:33.306 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:33.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:33.306 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:33.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:33.306 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:33.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:33.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:33.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:33.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:33.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:33.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:33.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:33.307 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:33.275+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:33.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:33.304+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:33.305012+0000) 2024-02-20T09:11:33.868 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:33.800+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:33.801255+0000) 2024-02-20T09:11:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:34.030+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:34.031477+0000) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:34.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:34.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:34.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:34.056+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:33.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:33.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:33.866+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:34.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:33.867+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:34.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:34.307+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:34.308235+0000) 2024-02-20T09:11:34.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:34.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:34.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:34.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:34.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:34.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:34.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:34.275+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:34.881 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:34.802+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:34.803590+0000) 2024-02-20T09:11:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:35.033+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.034088+0000) 2024-02-20T09:11:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:35.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:35.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:35.093+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:35.094+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:35.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:34.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:35.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:34.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:34.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:35.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:34.879+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:35.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:35.310+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.311798+0000) 2024-02-20T09:11:35.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:35.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:35.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:35.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:35.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:35.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:35.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:35.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:35.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:35.803+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.804203+0000) 2024-02-20T09:11:36.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:35.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:35.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:35.917+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:35.918+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:36.033+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.034534+0000) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:36.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:36.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:36.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:36.070+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:36.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:36.071+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:36.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:36.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:36.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:36.216+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:36.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:36.217+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:36.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:36.311+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.312518+0000) 2024-02-20T09:11:36.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:36.537+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57920 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:36.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:36.535+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.536024+0000) 2024-02-20T09:11:36.806 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:36.534+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.535433+0000) 2024-02-20T09:11:36.806 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:36.536+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.537265+0000) 2024-02-20T09:11:37.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:36.803+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.804407+0000) 2024-02-20T09:11:37.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:36.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:37.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:36.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:37.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:36.927+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:37.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:36.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:37.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:37.033+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:37.034725+0000) 2024-02-20T09:11:37.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:37.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:37.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:37.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:37.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:37.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:37.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:37.068+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:37.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:37.311+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:37.312695+0000) 2024-02-20T09:11:37.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:37.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:37.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:37.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:37.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:37.197+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:37.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:37.198+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:37.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:37.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:37.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:37.957+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:38.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:37.806+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:37.807286+0000) 2024-02-20T09:11:38.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:38.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:38.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:38.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:38.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:38.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:38.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.036+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:38.037154+0000) 2024-02-20T09:11:38.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:38.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:38.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:38.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:38.315+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:38.316124+0000) 2024-02-20T09:11:38.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:38.188+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:38.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:38.808+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:38.809429+0000) 2024-02-20T09:11:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:39.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:39.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:39.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:39.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:38.919+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:39.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:38.919+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:38.919+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:38.920+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:38.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:39.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:39.038+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:39.039632+0000) 2024-02-20T09:11:39.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:39.317+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:39.318700+0000) 2024-02-20T09:11:39.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:39.810+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:39.811720+0000) 2024-02-20T09:11:40.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:39.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:40.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:39.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:40.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:39.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:40.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:39.900+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:40.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:40.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:40.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:40.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:40.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:40.020+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:40.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:40.021+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:40.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:40.042+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.043013+0000) 2024-02-20T09:11:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:40.321+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.322266+0000) 2024-02-20T09:11:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:40.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:40.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:40.177+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:40.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:40.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:40.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:40.811+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.812194+0000) 2024-02-20T09:11:41.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:40.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:41.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:40.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:40.925+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:40.926+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.029+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.042+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.043478+0000) 2024-02-20T09:11:41.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:41.322+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.322955+0000) 2024-02-20T09:11:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:41.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:41.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:41.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:41.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:41.219+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:41.813 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:41.544+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57959 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:41.814 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:41.541+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.542924+0000) 2024-02-20T09:11:41.814 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.542+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.543071+0000) 2024-02-20T09:11:41.814 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:41.542+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.543198+0000) 2024-02-20T09:11:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:41.811+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.812364+0000) 2024-02-20T09:11:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:41.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:41.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:41.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:41.958+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:41.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:42.042+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:42.043650+0000) 2024-02-20T09:11:42.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:42.322+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:42.323129+0000) 2024-02-20T09:11:42.583 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:42.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:42.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:42.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:42.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:42.258+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:42.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:42.259+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:42.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:42.814+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:42.815332+0000) 2024-02-20T09:11:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:42.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:42.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:42.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:42.979+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:42.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:42.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:42.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:42.963+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:43.046+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:43.046987+0000) 2024-02-20T09:11:43.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:43.324+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:43.325647+0000) 2024-02-20T09:11:43.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:43.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:43.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:43.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:43.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:43.236+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:43.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:43.237+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:43.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:43.817+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:43.818911+0000) 2024-02-20T09:11:44.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:43.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:43.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:43.990+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:43.991+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:43.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:43.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:43.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:43.954+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:44.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:44.048+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:44.049744+0000) 2024-02-20T09:11:44.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:44.327+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:44.328243+0000) 2024-02-20T09:11:44.575 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:44.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:44.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:44.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:44.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:44.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:44.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:44.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:44.828 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:44.820+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:44.821672+0000) 2024-02-20T09:11:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:44.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:44.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:44.998+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:44.998+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:44.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:44.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:44.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:44.989+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:45.051+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.052338+0000) 2024-02-20T09:11:45.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:45.330+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.331051+0000) 2024-02-20T09:11:45.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:45.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:45.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:45.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:45.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:45.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:45.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:45.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:45.821+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.822191+0000) 2024-02-20T09:11:46.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:46.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:46.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:46.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.027+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:46.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:45.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:46.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:45.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:46.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:45.995+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:46.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:45.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:46.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:46.051+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.052788+0000) 2024-02-20T09:11:46.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.330+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.331741+0000) 2024-02-20T09:11:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:46.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:46.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:46.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:46.340+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:46.823 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:46.552+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 57998 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:46.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.548+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.549591+0000) 2024-02-20T09:11:46.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:46.549+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.550628+0000) 2024-02-20T09:11:46.824 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:46.549+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.550510+0000) 2024-02-20T09:11:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:46.821+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.822395+0000) 2024-02-20T09:11:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.984+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:46.985+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:47.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:47.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:47.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:47.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:47.052+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:47.052953+0000) 2024-02-20T09:11:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:47.330+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:47.331907+0000) 2024-02-20T09:11:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:47.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:47.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:47.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:47.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:47.345+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:47.824+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:47.825162+0000) 2024-02-20T09:11:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:47.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:47.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:47.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:47.990+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:48.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:48.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:48.001+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:48.002+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:48.054+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:48.055633+0000) 2024-02-20T09:11:48.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:48.333+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:48.334533+0000) 2024-02-20T09:11:48.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:48.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:48.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:48.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:48.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:48.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:48.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:48.337+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:48.826+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:48.827528+0000) 2024-02-20T09:11:49.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:48.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:49.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:48.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:48.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:48.982+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.016+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:49.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.058+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:49.058974+0000) 2024-02-20T09:11:49.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:49.335+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:49.336837+0000) 2024-02-20T09:11:49.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:49.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:49.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:49.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:49.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:49.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:49.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:49.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:49.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:49.828+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:49.829589+0000) 2024-02-20T09:11:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:49.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:49.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:49.950+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:49.951+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:49.984+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:50.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:50.061+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.062549+0000) 2024-02-20T09:11:50.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:50.338+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.339065+0000) 2024-02-20T09:11:50.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:50.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:50.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:50.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:50.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:50.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:50.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:50.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:50.829+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.830281+0000) 2024-02-20T09:11:51.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:50.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:51.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:50.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:51.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:50.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:51.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:50.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:51.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:51.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:51.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:51.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:51.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:51.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:51.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:51.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:51.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:51.062+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.063165+0000) 2024-02-20T09:11:51.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:51.338+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.339555+0000) 2024-02-20T09:11:51.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:51.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:51.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:51.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:51.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:51.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:51.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:51.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:51.560+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58038 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:51.829 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:51.558+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.559763+0000) 2024-02-20T09:11:51.829 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:51.556+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.557450+0000) 2024-02-20T09:11:51.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:51.557+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.558711+0000) 2024-02-20T09:11:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:51.829+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.830449+0000) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:51.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:51.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:51.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:51.969+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.002+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.002+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.002+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:52.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.003+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:52.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.062+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:52.063357+0000) 2024-02-20T09:11:52.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:52.338+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:52.339734+0000) 2024-02-20T09:11:52.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:52.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:52.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:52.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:52.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:52.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:52.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:52.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:52.832+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:52.833726+0000) 2024-02-20T09:11:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:52.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:52.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:52.967+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:52.968+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:52.982+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:53.064+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:53.065596+0000) 2024-02-20T09:11:53.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:53.341+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:53.342494+0000) 2024-02-20T09:11:53.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:53.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:53.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:53.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:53.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:53.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:53.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:53.329+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:53.834+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:53.835891+0000) 2024-02-20T09:11:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:53.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:53.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:53.959+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:53.959+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:53.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:53.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:53.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:53.957+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:54.067+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:54.068249+0000) 2024-02-20T09:11:54.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:54.343+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:54.344597+0000) 2024-02-20T09:11:54.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:54.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:54.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:54.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:54.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:54.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:54.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:54.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:55.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:54.837+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:54.838303+0000) 2024-02-20T09:11:55.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:54.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:55.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:54.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:55.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:54.934+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:55.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:54.935+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:55.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:54.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:55.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:54.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:54.953+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:54.954+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:55.226 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:55.069+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.070705+0000) 2024-02-20T09:11:55.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:55.347+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.347964+0000) 2024-02-20T09:11:55.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:55.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:55.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:55.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:55.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:55.360+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:55.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:55.361+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:55.838+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.839019+0000) 2024-02-20T09:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:55.916+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:55.916+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:55.916+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:55.916+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:56.070+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.071418+0000) 2024-02-20T09:11:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:55.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:55.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:55.906+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:55.907+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:56.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:56.347+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.348441+0000) 2024-02-20T09:11:56.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:56.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:56.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:56.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:56.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:56.401+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:56.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:56.401+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:56.829 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:11:56.567+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58078 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:11:56.829 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:56.564+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.565300+0000) 2024-02-20T09:11:56.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:56.564+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.565589+0000) 2024-02-20T09:11:56.830 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:56.563+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.564737+0000) 2024-02-20T09:11:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:56.838+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.839207+0000) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:56.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:56.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:56.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:56.917+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:56.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:56.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:56.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:56.888+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:57.070+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:57.071625+0000) 2024-02-20T09:11:57.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:57.347+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:57.348624+0000) 2024-02-20T09:11:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:57.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:57.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:57.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:57.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:57.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:57.840+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:57.841204+0000) 2024-02-20T09:11:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:57.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:57.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:57.901+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:57.902+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:57.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:57.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:57.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:57.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:58.074+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:58.075196+0000) 2024-02-20T09:11:58.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:58.349+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:58.350712+0000) 2024-02-20T09:11:58.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:58.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:58.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:58.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:58.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:58.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:58.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:58.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:58.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:58.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:58.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:58.914+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:58.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:58.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:58.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:11:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:58.948+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:59.077+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:59.078107+0000) 2024-02-20T09:11:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:58.844+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:58.845007+0000) 2024-02-20T09:11:59.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:59.352+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:59.353688+0000) 2024-02-20T09:11:59.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:59.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:59.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:59.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:59.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:59.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:11:59.586 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:59.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:59.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:59.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:59.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:11:59.871+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:59.968+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:59.968+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:59.968+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:11:59.970+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:00.079+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.080842+0000) 2024-02-20T09:12:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:11:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:11:59.847+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:59.848607+0000) 2024-02-20T09:12:00.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:00.355+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.356252+0000) 2024-02-20T09:12:00.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:00.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:00.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:00.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:00.586 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:00.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:00.586 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:00.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:00.848+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.849145+0000) 2024-02-20T09:12:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:00.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:00.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:00.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:00.928+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:01.080+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.081540+0000) 2024-02-20T09:12:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:00.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:00.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:00.865+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:01.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:00.866+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:01.570 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:01.356+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.356983+0000) 2024-02-20T09:12:01.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:01.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:01.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:01.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:01.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:01.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:01.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:01.248+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:01.829 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:01.573+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.574179+0000) 2024-02-20T09:12:01.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:01.573+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.574290+0000) 2024-02-20T09:12:01.830 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:01.576+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58118 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:01.830 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:01.572+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.573623+0000) 2024-02-20T09:12:02.082 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:01.848+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.849328+0000) 2024-02-20T09:12:02.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:01.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:02.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:01.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:02.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:01.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:02.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:01.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:02.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:01.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:02.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:01.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:02.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:01.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:02.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:01.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:02.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:02.080+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:02.081789+0000) 2024-02-20T09:12:02.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:02.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:02.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:02.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:02.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:02.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:02.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:02.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:02.356+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:02.357152+0000) 2024-02-20T09:12:03.086 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:02.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:03.086 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:02.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:03.086 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:02.870+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:03.086 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:02.871+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:03.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:02.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:03.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:02.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:03.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:02.977+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:03.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:02.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:03.087 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:02.850+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:02.851727+0000) 2024-02-20T09:12:03.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:03.083+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:03.084848+0000) 2024-02-20T09:12:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:03.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:03.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:03.286+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:03.287+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:03.359+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:03.359976+0000) 2024-02-20T09:12:04.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:03.853+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:03.854180+0000) 2024-02-20T09:12:04.088 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:03.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:04.088 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:03.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:04.088 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:03.904+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:04.088 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:03.905+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:04.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:03.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:04.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:03.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:04.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:03.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:04.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:03.972+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:04.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:04.085+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:04.086909+0000) 2024-02-20T09:12:04.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:04.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:04.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:04.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:04.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:04.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:04.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:04.250+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:04.362+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:04.363545+0000) 2024-02-20T09:12:05.091 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:04.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:05.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:04.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:05.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:04.929+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:05.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:04.930+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:05.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:04.855+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:04.856766+0000) 2024-02-20T09:12:05.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:04.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:05.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:04.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:05.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:04.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:05.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:04.958+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:05.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:05.089+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.090495+0000) 2024-02-20T09:12:05.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:05.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:05.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:05.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:05.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:05.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:05.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:05.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:05.364+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.365773+0000) 2024-02-20T09:12:06.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:05.856+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.857486+0000) 2024-02-20T09:12:06.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:05.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:06.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:05.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:05.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:05.895+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:05.971+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:05.971+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:05.971+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:05.972+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:06.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:06.090+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.091117+0000) 2024-02-20T09:12:06.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:06.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:06.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:06.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:06.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:06.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:06.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:06.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:06.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:06.365+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.366411+0000) 2024-02-20T09:12:06.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:06.580+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.581404+0000) 2024-02-20T09:12:06.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:06.580+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.581249+0000) 2024-02-20T09:12:06.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:06.580+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.581867+0000) 2024-02-20T09:12:06.706 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:06.583+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58157 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:06.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:06.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:06.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:06.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:06.856+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.857658+0000) 2024-02-20T09:12:07.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:06.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:07.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:06.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:07.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:06.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:07.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:06.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:07.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:07.090+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:07.091324+0000) 2024-02-20T09:12:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:07.365+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:07.366602+0000) 2024-02-20T09:12:07.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:07.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:07.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:07.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:07.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:07.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:08.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:07.858+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:07.859724+0000) 2024-02-20T09:12:08.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:07.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:08.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:07.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:08.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:07.985+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:08.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:07.986+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:08.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:07.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:08.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:07.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:08.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:07.900+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:08.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:07.901+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:08.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:08.092+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:08.093846+0000) 2024-02-20T09:12:08.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:08.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:08.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:08.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:08.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:08.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:08.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:08.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:08.369+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:08.370238+0000) 2024-02-20T09:12:09.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:08.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:09.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:08.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:09.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:08.903+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:09.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:08.904+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:09.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:09.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:09.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.018+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:09.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.019+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:09.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:08.861+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:08.862736+0000) 2024-02-20T09:12:09.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.094+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:09.096112+0000) 2024-02-20T09:12:09.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:09.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:09.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:09.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:09.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:09.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:09.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:09.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:09.370+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:09.372819+0000) 2024-02-20T09:12:10.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:09.863+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:09.865736+0000) 2024-02-20T09:12:10.100 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:09.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:10.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:09.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:10.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:09.920+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:10.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:09.921+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:10.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:10.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:10.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:10.102 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:09.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:10.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:10.097+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.099059+0000) 2024-02-20T09:12:10.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:10.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:10.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:10.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:10.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:10.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:10.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:10.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:10.374+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.376465+0000) 2024-02-20T09:12:11.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:10.864+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.866530+0000) 2024-02-20T09:12:11.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:10.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:11.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:10.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:11.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:10.886+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:11.101 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:10.887+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:11.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:10.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:11.102 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:10.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:11.102 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:10.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:11.102 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:10.992+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:11.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:11.097+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.099724+0000) 2024-02-20T09:12:11.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:11.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:11.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:11.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:11.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:11.315+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:11.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:11.316+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:11.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:11.589+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58197 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:11.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:11.587+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.589152+0000) 2024-02-20T09:12:11.612 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:11.586+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.588783+0000) 2024-02-20T09:12:11.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:11.375+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.377259+0000) 2024-02-20T09:12:11.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:11.587+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.589443+0000) 2024-02-20T09:12:11.868 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:11.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:11.868 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:11.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:11.868 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:11.853+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:11.868 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:11.853+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:11.865+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.866707+0000) 2024-02-20T09:12:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:11.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:11.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:11.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:11.959+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:12.098+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:12.100102+0000) 2024-02-20T09:12:12.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:12.376+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:12.377454+0000) 2024-02-20T09:12:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:12.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:12.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:12.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:12.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:13.104 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:12.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:13.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:12.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:13.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:12.878+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:13.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:12.879+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:13.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:12.868+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:12.869082+0000) 2024-02-20T09:12:13.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:12.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:13.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:12.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:13.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:12.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:13.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:12.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:13.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:13.102+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:13.103635+0000) 2024-02-20T09:12:13.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:13.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:13.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:13.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:13.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:13.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:13.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:13.289+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:13.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:13.379+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:13.380498+0000) 2024-02-20T09:12:14.108 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:13.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:14.108 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:13.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:14.109 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:13.894+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:14.109 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:13.895+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:14.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:13.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:14.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:13.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:14.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:13.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:14.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:13.950+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:14.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:13.870+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:13.871634+0000) 2024-02-20T09:12:14.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:14.106+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:14.107188+0000) 2024-02-20T09:12:14.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:14.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:14.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:14.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:14.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:14.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:14.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:14.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:14.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:14.382+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:14.383089+0000) 2024-02-20T09:12:15.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:14.873+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:14.874138+0000) 2024-02-20T09:12:15.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:14.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:15.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:14.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:15.112 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:14.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:15.112 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:14.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:15.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:14.921+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:15.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:14.921+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:15.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:14.921+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:15.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:14.922+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:15.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:15.109+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.110154+0000) 2024-02-20T09:12:15.387 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:15.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:15.387 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:15.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:15.387 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:15.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:15.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:15.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:15.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:15.385+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.385955+0000) 2024-02-20T09:12:16.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:15.873+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.874683+0000) 2024-02-20T09:12:16.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:15.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:16.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:15.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:16.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:15.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:16.113 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:15.959+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:16.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:15.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:16.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:15.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:16.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:15.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:16.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:15.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:16.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:16.109+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.110829+0000) 2024-02-20T09:12:16.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:16.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:16.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:16.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:16.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:16.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:16.388 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:16.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:16.719 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:16.598+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58236 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:16.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:16.594+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.595237+0000) 2024-02-20T09:12:16.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:16.596+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.597000+0000) 2024-02-20T09:12:16.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:16.385+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.386646+0000) 2024-02-20T09:12:16.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:16.595+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.596798+0000) 2024-02-20T09:12:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:16.873+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.874921+0000) 2024-02-20T09:12:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:16.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:16.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:16.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:16.859+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:17.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:17.110+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:17.111044+0000) 2024-02-20T09:12:17.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:16.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:17.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:16.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:17.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:16.979+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:17.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:16.980+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:17.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:17.385+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:17.386880+0000) 2024-02-20T09:12:17.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:17.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:17.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:17.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:17.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:17.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:17.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:17.313+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:18.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:17.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:18.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:17.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:18.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:17.971+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:18.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:17.972+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:18.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:17.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:18.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:17.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:18.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:17.904+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:18.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:17.905+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:18.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:17.876+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:17.877242+0000) 2024-02-20T09:12:18.390 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:18.112+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:18.113666+0000) 2024-02-20T09:12:18.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:18.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:18.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:18.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:18.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:18.293+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:18.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:18.294+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:18.388+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:18.389472+0000) 2024-02-20T09:12:19.117 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:18.878+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:18.879884+0000) 2024-02-20T09:12:19.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:18.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:19.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:18.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:19.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:18.955+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:19.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:18.956+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:19.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:18.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:19.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:18.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:19.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:18.927+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:19.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:18.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:19.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:19.115+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:19.116557+0000) 2024-02-20T09:12:19.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:19.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:19.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:19.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:19.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:19.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:19.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:19.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:19.391+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:19.392400+0000) 2024-02-20T09:12:20.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:19.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:20.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:19.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:20.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:19.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:20.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:20.000+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:20.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:19.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:20.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:19.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:20.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:19.976+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:20.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:19.977+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:20.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:19.881+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:19.882381+0000) 2024-02-20T09:12:20.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:20.118+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.119337+0000) 2024-02-20T09:12:20.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:20.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:20.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:20.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:20.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:20.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:20.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:20.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:20.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:20.393+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.394507+0000) 2024-02-20T09:12:21.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:20.881+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.882896+0000) 2024-02-20T09:12:21.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:20.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:21.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:20.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:21.122 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:20.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:21.122 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:20.942+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:21.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:21.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:21.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:21.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:21.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:21.044+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:21.122 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:21.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:21.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:21.119+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.120189+0000) 2024-02-20T09:12:21.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:21.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:21.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:21.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:21.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:21.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:21.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:21.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:21.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:21.605+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58276 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:21.602+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.603227+0000) 2024-02-20T09:12:21.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:21.602+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.603736+0000) 2024-02-20T09:12:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:21.394+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.395063+0000) 2024-02-20T09:12:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:21.602+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.603369+0000) 2024-02-20T09:12:22.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:21.971+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:22.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:21.971+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:22.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:21.971+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:22.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:21.972+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:22.055 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:21.882+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.883076+0000) 2024-02-20T09:12:22.336 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:22.119+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:22.120386+0000) 2024-02-20T09:12:22.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:22.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:22.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:22.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:22.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:22.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:22.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:22.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:22.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:22.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:22.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:22.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:22.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:22.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:22.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:22.209+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:22.394+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:22.395251+0000) 2024-02-20T09:12:23.123 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:22.884+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:22.885385+0000) 2024-02-20T09:12:23.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:23.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:23.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:23.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:23.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:23.056+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:23.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:23.057+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:23.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:22.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:23.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:22.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:23.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:22.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:23.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:22.944+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:23.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:23.121+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:23.122791+0000) 2024-02-20T09:12:23.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:23.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:23.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:23.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:23.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:23.255+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:23.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:23.256+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:23.396+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:23.397875+0000) 2024-02-20T09:12:24.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:24.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:24.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:24.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:24.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:24.039+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:24.127 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:24.039+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:24.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:23.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:24.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:23.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:24.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:23.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:24.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:23.915+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:24.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:23.887+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:23.888164+0000) 2024-02-20T09:12:24.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:24.124+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:24.125215+0000) 2024-02-20T09:12:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:24.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:24.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:24.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:24.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:24.289+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:24.399+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:24.400379+0000) 2024-02-20T09:12:25.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:24.889+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:24.890658+0000) 2024-02-20T09:12:25.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:24.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:25.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:24.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:25.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:24.868+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:25.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:24.869+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:25.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:25.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:25.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:25.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:25.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:25.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:25.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:25.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:25.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:25.126+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.127868+0000) 2024-02-20T09:12:25.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:25.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:25.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:25.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:25.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:25.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:25.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:25.302+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:25.401+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.402679+0000) 2024-02-20T09:12:26.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:25.890+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.891570+0000) 2024-02-20T09:12:26.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:26.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:26.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:26.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:26.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:26.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:26.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:26.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:26.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:25.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:26.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:25.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:26.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:25.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:26.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:25.827+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:26.391 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:26.127+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.128633+0000) 2024-02-20T09:12:26.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:26.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:26.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:26.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:26.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:26.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:26.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:26.259+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:26.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:26.613+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58315 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:26.611+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.612867+0000) 2024-02-20T09:12:26.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:26.610+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.611407+0000) 2024-02-20T09:12:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:26.402+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.403683+0000) 2024-02-20T09:12:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:26.611+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.612080+0000) 2024-02-20T09:12:27.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:26.890+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.891784+0000) 2024-02-20T09:12:27.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:26.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:27.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:26.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:27.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:26.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:27.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:26.787+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:27.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:27.127+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:27.128872+0000) 2024-02-20T09:12:27.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:27.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:27.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:27.055+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:27.056+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:27.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:27.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:27.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:27.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:27.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:27.402+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:27.403885+0000) 2024-02-20T09:12:28.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:27.893+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:27.893969+0000) 2024-02-20T09:12:28.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:27.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:28.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:27.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:28.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:27.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:28.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:27.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:28.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:28.131+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:28.132393+0000) 2024-02-20T09:12:28.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:28.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:28.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:28.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:28.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:28.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:28.299 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:28.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:28.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:28.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:28.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:28.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:28.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:28.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:28.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:28.269+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:28.405+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:28.406311+0000) 2024-02-20T09:12:29.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:28.896+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:28.897521+0000) 2024-02-20T09:12:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:28.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:28.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:28.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:29.037 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:28.785+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:29.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:29.134+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:29.135736+0000) 2024-02-20T09:12:29.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:29.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:29.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:29.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:29.300 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:29.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:29.301 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:29.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:29.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:29.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:29.260+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:29.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:29.407+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:29.408857+0000) 2024-02-20T09:12:30.081 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:29.898+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:29.899818+0000) 2024-02-20T09:12:30.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:29.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:30.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:29.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:30.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:29.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:30.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:29.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:30.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:30.137+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.138303+0000) 2024-02-20T09:12:30.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:30.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:30.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:30.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:30.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:30.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:30.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:30.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:30.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:30.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:30.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:30.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:30.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:30.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:30.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:30.275+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:30.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:30.410+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.410965+0000) 2024-02-20T09:12:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:30.899+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.900487+0000) 2024-02-20T09:12:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:30.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:30.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:30.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:30.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:31.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:31.137+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.138866+0000) 2024-02-20T09:12:31.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:31.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:31.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:31.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:31.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:31.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:31.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:31.119+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:31.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:31.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:31.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:31.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:31.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:31.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:31.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:31.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:31.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:31.620+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58355 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:31.617+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.618307+0000) 2024-02-20T09:12:31.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:31.619+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.620846+0000) 2024-02-20T09:12:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:31.410+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.411817+0000) 2024-02-20T09:12:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:31.617+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.618915+0000) 2024-02-20T09:12:32.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:31.899+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.900674+0000) 2024-02-20T09:12:32.089 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:31.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:32.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:31.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:32.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:31.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:32.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:31.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:32.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:32.138+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:32.139059+0000) 2024-02-20T09:12:32.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:32.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:32.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:32.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:32.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:32.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:32.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:32.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:32.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:32.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:32.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:32.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:32.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:32.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:32.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:32.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:32.411+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:32.411965+0000) 2024-02-20T09:12:33.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:32.903+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:32.904015+0000) 2024-02-20T09:12:33.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:32.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:33.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:32.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:33.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:32.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:33.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:32.750+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:33.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:33.140+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:33.141862+0000) 2024-02-20T09:12:33.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:33.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:33.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:33.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:33.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:33.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:33.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:33.138+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:33.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:33.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:33.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:33.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:33.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:33.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:33.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:33.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:33.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:33.414+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:33.415503+0000) 2024-02-20T09:12:34.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:33.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:34.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:33.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:34.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:33.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:34.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:33.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:34.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:33.906+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:33.906976+0000) 2024-02-20T09:12:34.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:34.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:34.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:34.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:34.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:34.153+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:34.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:34.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:34.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:34.144+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:34.145448+0000) 2024-02-20T09:12:34.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:34.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:34.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:34.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:34.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:34.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:34.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:34.227+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:34.416+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:34.417551+0000) 2024-02-20T09:12:35.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:34.908+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:34.909734+0000) 2024-02-20T09:12:35.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:34.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:35.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:34.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:35.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:34.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:35.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:34.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:35.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:35.148+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.148928+0000) 2024-02-20T09:12:35.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:35.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:35.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:35.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:35.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:35.273+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:35.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:35.274+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:35.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:35.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:35.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:35.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:35.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:35.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:35.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:35.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:35.419+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.420236+0000) 2024-02-20T09:12:36.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:35.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:36.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:35.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:36.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:35.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:36.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:35.803+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:36.097 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:35.909+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.910587+0000) 2024-02-20T09:12:36.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:36.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:36.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:36.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:36.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:36.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:36.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:36.095+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:36.361 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:36.148+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.149628+0000) 2024-02-20T09:12:36.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:36.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:36.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:36.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:36.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:36.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:36.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:36.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:36.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:36.420+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.421017+0000) 2024-02-20T09:12:36.911 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:36.627+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58394 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:36.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:36.625+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.626422+0000) 2024-02-20T09:12:36.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:36.624+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.625874+0000) 2024-02-20T09:12:36.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:36.624+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.625774+0000) 2024-02-20T09:12:36.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:36.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:36.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:36.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:36.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:36.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:36.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:36.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:37.148+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:37.149808+0000) 2024-02-20T09:12:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:36.909+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.910755+0000) 2024-02-20T09:12:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:37.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:37.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:37.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:37.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:37.420+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:37.421496+0000) 2024-02-20T09:12:37.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:37.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:37.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:37.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:37.327+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:38.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:37.913+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:37.914018+0000) 2024-02-20T09:12:38.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:37.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:38.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:37.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:38.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:37.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:38.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:37.891+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:38.425 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:38.152+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:38.153121+0000) 2024-02-20T09:12:38.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:38.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:38.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:38.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:38.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:38.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:38.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:38.149+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:38.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:38.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:38.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:38.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:38.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:38.362+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:38.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:38.364+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:38.423+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:38.424661+0000) 2024-02-20T09:12:39.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:38.915+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:38.916125+0000) 2024-02-20T09:12:39.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:38.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:39.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:38.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:39.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:38.876+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:39.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:38.877+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:39.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:39.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:39.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:39.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:39.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:39.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:39.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:39.193+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:39.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:39.155+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:39.155967+0000) 2024-02-20T09:12:39.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:39.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:39.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:39.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:39.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:39.321+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:39.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:39.321+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:39.426+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:39.427481+0000) 2024-02-20T09:12:40.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:39.917+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:39.918209+0000) 2024-02-20T09:12:40.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:39.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:40.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:39.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:40.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:39.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:40.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:39.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:40.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:40.157+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.158199+0000) 2024-02-20T09:12:40.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:40.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:40.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:40.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:40.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:40.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:40.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:40.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:40.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:40.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:40.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:40.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:40.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:40.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:40.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:40.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:40.429+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.430538+0000) 2024-02-20T09:12:41.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:40.917+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.918772+0000) 2024-02-20T09:12:41.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:40.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:41.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:40.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:41.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:40.860+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:41.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:40.861+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:41.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:41.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:41.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:41.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:41.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:41.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:41.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:41.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:41.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:41.158+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.159273+0000) 2024-02-20T09:12:41.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:41.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:41.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:41.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:41.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:41.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:41.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:41.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:41.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:41.634+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58434 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:41.635+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.636182+0000) 2024-02-20T09:12:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:41.632+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.633244+0000) 2024-02-20T09:12:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:41.430+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.431658+0000) 2024-02-20T09:12:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:41.632+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.633125+0000) 2024-02-20T09:12:42.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:41.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:42.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:41.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:42.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:41.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:42.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:41.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:42.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:41.918+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.918973+0000) 2024-02-20T09:12:42.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:42.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:42.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:42.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:42.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:42.279+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:42.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:42.280+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:42.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:42.158+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:42.159464+0000) 2024-02-20T09:12:42.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:42.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:42.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:42.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:42.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:42.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:42.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:42.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:42.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:42.430+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:42.431871+0000) 2024-02-20T09:12:43.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:42.919+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:42.920856+0000) 2024-02-20T09:12:43.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:42.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:43.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:42.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:43.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:42.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:43.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:42.781+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:43.437 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:43.161+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:43.162108+0000) 2024-02-20T09:12:43.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:43.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:43.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:43.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:43.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:43.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:43.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:43.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:43.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:43.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:43.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:43.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:43.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:43.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:43.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:43.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:43.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:43.433+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:43.434371+0000) 2024-02-20T09:12:44.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:43.922+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:43.923399+0000) 2024-02-20T09:12:44.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:43.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:44.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:43.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:44.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:43.809+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:44.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:43.809+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:44.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:44.163+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:44.164739+0000) 2024-02-20T09:12:44.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:44.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:44.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:44.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:44.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:44.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:44.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:44.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:44.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:44.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:44.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:44.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:44.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:44.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:44.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:44.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:44.435+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:44.436794+0000) 2024-02-20T09:12:45.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:44.925+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:44.926216+0000) 2024-02-20T09:12:45.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:44.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:45.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:44.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:45.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:44.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:45.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:44.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:45.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:45.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:45.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:45.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:45.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:45.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:45.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:45.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:45.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:45.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.167823+0000) 2024-02-20T09:12:45.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:45.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:45.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:45.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:45.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:45.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:45.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:45.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:45.439+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.439966+0000) 2024-02-20T09:12:46.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:45.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:46.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:45.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:46.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:45.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:46.016 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:45.748+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:46.016 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:45.925+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.926800+0000) 2024-02-20T09:12:46.273 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:46.167+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.168872+0000) 2024-02-20T09:12:46.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:46.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:46.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:46.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:46.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:46.234+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:46.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:46.235+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:46.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:46.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:46.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:46.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:46.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:46.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:46.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:46.272+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:46.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:46.439+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.440671+0000) 2024-02-20T09:12:46.928 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:46.641+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58473 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:46.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:46.638+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.639906+0000) 2024-02-20T09:12:46.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:46.639+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.640238+0000) 2024-02-20T09:12:46.929 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:46.639+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.640924+0000) 2024-02-20T09:12:46.929 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:46.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:46.929 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:46.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:46.929 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:46.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:46.929 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:46.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:47.168+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:47.169064+0000) 2024-02-20T09:12:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:46.926+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.926979+0000) 2024-02-20T09:12:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:47.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:47.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:47.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:47.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:47.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:47.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:47.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:47.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:47.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:47.439+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:47.440865+0000) 2024-02-20T09:12:47.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:47.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:47.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:47.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:47.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:47.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:47.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:47.729+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:48.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:47.929+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:47.930163+0000) 2024-02-20T09:12:48.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:48.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:48.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:48.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:48.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:48.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:48.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:48.358+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:48.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:48.170+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:48.171056+0000) 2024-02-20T09:12:48.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:48.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:48.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:48.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:48.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:48.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:48.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:48.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:48.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:48.443+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:48.444146+0000) 2024-02-20T09:12:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:48.932+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:48.933529+0000) 2024-02-20T09:12:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:48.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:48.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:48.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:48.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:49.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:49.172+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:49.173448+0000) 2024-02-20T09:12:49.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:49.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:49.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:49.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:49.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:49.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:49.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:49.333+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:49.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:49.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:49.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:49.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:49.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:49.252+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:49.449 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:49.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:49.446+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:49.447275+0000) 2024-02-20T09:12:50.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:49.935+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:49.936022+0000) 2024-02-20T09:12:50.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:49.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:50.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:49.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:50.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:49.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:50.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:49.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:50.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:50.175+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.175973+0000) 2024-02-20T09:12:50.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:50.345+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:50.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:50.345+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:50.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:50.345+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:50.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:50.346+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:50.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:50.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:50.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:50.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:50.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:50.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:50.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:50.244+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:50.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:50.448+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.449671+0000) 2024-02-20T09:12:51.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:50.936+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.936959+0000) 2024-02-20T09:12:51.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:50.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:51.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:50.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:51.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:50.728+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:51.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:50.729+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:51.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:51.175+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.176758+0000) 2024-02-20T09:12:51.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:51.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:51.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:51.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:51.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:51.319+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:51.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:51.320+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:51.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:51.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:51.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:51.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:51.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:51.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:51.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:51.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:51.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:51.649+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58513 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:51.645+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.646730+0000) 2024-02-20T09:12:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:51.646+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.647511+0000) 2024-02-20T09:12:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:51.449+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.450470+0000) 2024-02-20T09:12:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:51.646+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.647362+0000) 2024-02-20T09:12:52.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:51.936+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.937113+0000) 2024-02-20T09:12:52.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:51.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:52.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:51.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:52.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:51.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:52.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:51.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:52.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:52.176+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:52.176967+0000) 2024-02-20T09:12:52.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:52.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:52.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:52.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:52.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:52.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:52.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:52.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:52.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:52.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:52.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:52.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:52.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:52.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:52.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:52.302+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:52.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:52.449+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:52.450680+0000) 2024-02-20T09:12:53.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:52.938+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:52.939713+0000) 2024-02-20T09:12:53.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:53.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:53.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:52.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:53.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:52.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:53.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:52.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:53.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:52.749+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:53.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:53.178+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:53.179379+0000) 2024-02-20T09:12:53.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:53.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:53.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:53.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:53.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:53.274+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:53.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:53.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:53.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:53.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:53.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:53.161+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:53.455 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:53.162+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:53.452+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:53.453335+0000) 2024-02-20T09:12:54.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:53.940+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:53.941905+0000) 2024-02-20T09:12:54.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:53.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:54.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:53.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:54.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:53.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:54.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:53.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:54.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:54.181+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:54.182066+0000) 2024-02-20T09:12:54.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:54.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:54.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:54.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:54.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:54.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:54.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:54.291+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:54.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:54.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:54.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:54.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:54.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:54.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:54.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:54.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:54.454+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:54.455650+0000) 2024-02-20T09:12:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:54.943+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:54.943980+0000) 2024-02-20T09:12:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:54.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:54.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:54.719+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:54.720+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:55.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:55.184+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.185643+0000) 2024-02-20T09:12:55.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:55.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:55.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:55.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:55.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:55.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:55.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:55.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:55.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:55.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:55.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:55.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:55.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:55.457+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.458441+0000) 2024-02-20T09:12:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:55.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:55.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:55.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:55.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:56.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:55.943+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.944713+0000) 2024-02-20T09:12:56.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:56.185+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.186596+0000) 2024-02-20T09:12:56.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:56.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:56.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:56.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:56.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:56.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:56.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:56.176+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:56.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:56.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:56.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:56.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:56.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:56.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:56.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:56.304+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:56.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:12:56.656+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58552 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:12:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:56.653+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.654597+0000) 2024-02-20T09:12:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:56.653+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.654846+0000) 2024-02-20T09:12:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:56.458+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.459291+0000) 2024-02-20T09:12:56.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:56.653+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.654432+0000) 2024-02-20T09:12:57.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:56.943+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.944877+0000) 2024-02-20T09:12:57.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:56.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:57.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:56.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:57.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:56.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:57.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:56.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:57.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:57.185+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:57.186791+0000) 2024-02-20T09:12:57.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:57.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:57.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:57.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:57.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:57.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:57.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:57.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:57.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:57.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:57.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:57.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:57.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:57.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:57.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:57.165+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:57.458+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:57.459709+0000) 2024-02-20T09:12:58.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:57.946+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:57.947369+0000) 2024-02-20T09:12:58.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:58.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:58.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:58.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:58.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:57.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:58.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:57.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:58.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:57.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:58.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:57.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:58.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:58.188+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:58.188953+0000) 2024-02-20T09:12:58.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:58.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:58.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:58.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:58.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:58.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:58.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:58.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:58.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:58.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:58.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:58.133+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:58.461+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:58.462857+0000) 2024-02-20T09:12:59.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:58.948+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:58.949834+0000) 2024-02-20T09:12:59.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:59.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:59.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:59.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:59.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:58.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:59.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:58.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:59.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:58.801+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:12:59.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:58.802+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:59.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:59.191+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:59.192176+0000) 2024-02-20T09:12:59.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:59.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:12:59.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:59.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:59.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:59.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:59.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:59.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:12:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:59.358+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:12:59.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:12:59.465+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:59.466223+0000) 2024-02-20T09:13:00.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:12:59.952+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:59.953466+0000) 2024-02-20T09:13:00.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:00.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:00.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:00.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:00.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:00.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:00.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:00.077+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:00.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:59.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:00.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:59.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:00.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:59.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:00.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:12:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:12:59.842+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:00.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:00.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:00.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:00.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:00.395+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:00.194+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.195571+0000) 2024-02-20T09:13:00.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:00.467+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.468755+0000) 2024-02-20T09:13:01.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:00.953+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.954425+0000) 2024-02-20T09:13:01.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:01.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:01.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:01.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:01.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:00.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:01.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:00.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:01.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:00.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:01.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:00.817+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:01.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:01.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:01.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:01.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:01.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:01.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:01.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:01.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:01.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:01.195+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.196417+0000) 2024-02-20T09:13:01.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:01.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:01.377 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:01.111+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:01.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:01.468+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.469531+0000) 2024-02-20T09:13:01.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:01.661+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.662767+0000) 2024-02-20T09:13:01.915 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:01.661+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.662493+0000) 2024-02-20T09:13:01.915 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:01.815+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:01.915 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:01.815+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:01.915 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:01.815+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:01.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:01.816+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:01.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:01.660+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.661916+0000) 2024-02-20T09:13:01.916 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:01.663+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58592 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:02.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:01.952+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.954644+0000) 2024-02-20T09:13:02.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:02.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:02.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:02.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:02.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:02.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:02.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:02.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:02.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:02.194+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:02.196637+0000) 2024-02-20T09:13:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:02.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:02.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:02.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:02.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:02.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:02.467+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:02.469714+0000) 2024-02-20T09:13:03.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:02.954+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:02.956733+0000) 2024-02-20T09:13:03.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:02.827+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:03.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:02.827+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:03.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:02.827+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:03.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:02.828+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:03.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:03.197+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:03.199633+0000) 2024-02-20T09:13:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:03.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:03.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:03.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:03.188+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:03.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:03.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:03.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:03.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:03.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:03.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:03.471+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:03.472952+0000) 2024-02-20T09:13:04.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:03.958+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:03.960354+0000) 2024-02-20T09:13:04.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:04.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:04.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:04.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:04.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:04.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:04.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:03.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:04.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:03.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:04.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:03.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:04.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:03.811+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:04.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:04.199+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:04.201927+0000) 2024-02-20T09:13:04.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:04.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:04.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:04.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:04.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:04.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:04.448 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:04.364+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:04.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:04.179+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:04.474+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:04.476515+0000) 2024-02-20T09:13:05.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:04.961+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:04.963743+0000) 2024-02-20T09:13:05.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:04.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:05.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:04.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:05.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:04.816+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:05.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:04.817+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:05.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:05.202+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.204796+0000) 2024-02-20T09:13:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:05.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:05.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:05.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:05.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:05.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:05.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:05.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:05.220+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:05.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:05.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:05.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:05.479+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.480110+0000) 2024-02-20T09:13:06.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:05.963+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.964752+0000) 2024-02-20T09:13:06.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:05.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:06.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:05.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:06.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:05.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:06.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:05.866+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:06.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:06.204+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.205633+0000) 2024-02-20T09:13:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:06.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:06.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:06.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:06.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:06.243+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:06.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:06.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:06.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:06.363+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:06.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:06.363+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:06.791 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:06.670+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58630 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:06.668+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.669397+0000) 2024-02-20T09:13:06.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:06.667+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.668847+0000) 2024-02-20T09:13:06.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:06.480+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.481070+0000) 2024-02-20T09:13:06.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:06.667+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.668642+0000) 2024-02-20T09:13:07.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:06.964+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.964976+0000) 2024-02-20T09:13:07.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:06.840+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:07.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:06.840+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:07.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:06.840+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:07.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:06.841+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:07.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:07.204+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:07.205788+0000) 2024-02-20T09:13:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:07.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:07.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:07.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:07.388+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:07.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:07.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:07.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:07.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:07.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:07.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:07.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:07.480+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:07.481260+0000) 2024-02-20T09:13:08.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:07.966+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:07.967189+0000) 2024-02-20T09:13:08.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:07.851+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:08.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:07.851+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:08.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:07.851+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:08.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:07.852+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:08.207+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:08.208178+0000) 2024-02-20T09:13:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:08.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:08.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:08.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:08.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:08.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:08.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:08.412+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:08.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:08.413+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:08.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:08.483+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:08.484494+0000) 2024-02-20T09:13:09.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:08.969+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:08.970622+0000) 2024-02-20T09:13:09.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:08.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:09.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:08.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:09.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:08.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:09.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:08.891+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:09.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:09.209+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:09.210744+0000) 2024-02-20T09:13:09.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:09.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:09.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:09.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:09.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:09.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:09.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:09.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:09.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:09.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:09.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:09.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:09.423+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:09.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:09.486+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:09.487922+0000) 2024-02-20T09:13:10.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:09.972+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:09.973014+0000) 2024-02-20T09:13:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:09.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:09.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:09.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:09.890+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:10.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.212971+0000) 2024-02-20T09:13:10.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:10.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:10.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:10.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:10.417+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:10.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:10.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:10.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:10.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:10.307+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:10.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:10.490+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.491469+0000) 2024-02-20T09:13:11.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:10.973+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.973966+0000) 2024-02-20T09:13:11.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:10.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:11.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:10.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:11.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:10.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:11.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:10.890+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:11.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.213820+0000) 2024-02-20T09:13:11.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:11.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:11.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:11.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:11.305+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:11.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:11.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:11.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:11.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:11.421+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:11.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:11.677+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58662 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:11.799 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:11.676+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.676970+0000) 2024-02-20T09:13:11.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:11.674+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.675906+0000) 2024-02-20T09:13:11.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:11.491+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.492472+0000) 2024-02-20T09:13:11.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:11.677+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.678327+0000) 2024-02-20T09:13:12.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:11.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:12.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:11.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:12.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:11.875+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:12.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:11.876+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:12.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:11.973+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.974138+0000) 2024-02-20T09:13:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:12.213+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:12.214020+0000) 2024-02-20T09:13:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:12.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:12.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:12.378+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:12.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:12.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:12.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:12.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:12.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:12.836 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:12.491+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:12.492646+0000) 2024-02-20T09:13:13.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:12.975+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:12.976705+0000) 2024-02-20T09:13:13.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:12.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:13.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:12.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:13.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:12.864+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:13.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:12.865+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:13.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:13.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:13.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:13.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:13.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:13.215+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:13.216780+0000) 2024-02-20T09:13:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:13.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:13.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:13.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:13.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:13.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:13.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:13.494+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:13.495140+0000) 2024-02-20T09:13:14.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:13.979+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:13.979972+0000) 2024-02-20T09:13:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:13.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:14.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:13.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:13.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:13.880+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:14.218+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:14.219164+0000) 2024-02-20T09:13:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:14.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:14.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:14.394+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:14.395+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:14.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:14.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:14.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:14.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:14.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:14.281+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:14.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:14.496+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:14.497772+0000) 2024-02-20T09:13:15.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:14.981+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:14.982729+0000) 2024-02-20T09:13:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:14.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:14.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:14.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:14.892+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:15.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:15.220+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.221646+0000) 2024-02-20T09:13:15.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:15.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:15.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:15.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:15.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:15.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:15.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:15.411+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:15.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:15.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:15.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:15.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:15.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:15.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:15.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:15.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:15.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:15.499+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.500105+0000) 2024-02-20T09:13:16.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:15.982+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.983320+0000) 2024-02-20T09:13:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:15.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:15.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:15.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:15.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:16.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:16.221+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.222641+0000) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:16.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:16.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:16.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:16.385+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:16.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:16.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:16.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:16.270+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:16.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:16.271+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:16.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:16.684+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58701 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:16.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:16.682+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.683781+0000) 2024-02-20T09:13:16.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:16.682+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.683620+0000) 2024-02-20T09:13:16.805 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:16.500+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.501456+0000) 2024-02-20T09:13:16.805 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:16.683+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.684272+0000) 2024-02-20T09:13:17.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:16.982+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.983516+0000) 2024-02-20T09:13:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:16.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:16.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:16.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:16.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:17.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:17.221+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:17.222842+0000) 2024-02-20T09:13:17.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:17.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:17.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:17.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:17.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:17.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:17.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:17.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:17.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:17.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:17.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:17.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:17.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:17.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:17.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:17.300+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:17.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:17.500+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:17.501661+0000) 2024-02-20T09:13:18.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:17.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:17.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:17.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:17.918+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:17.984+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:17.985743+0000) 2024-02-20T09:13:18.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:18.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:18.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:18.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:18.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:18.428+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:18.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:18.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:18.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:18.225+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:18.226403+0000) 2024-02-20T09:13:18.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:18.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:18.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:18.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:18.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:18.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:18.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:18.347+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:18.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:18.502+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:18.503912+0000) 2024-02-20T09:13:19.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:18.987+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:18.988363+0000) 2024-02-20T09:13:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:18.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:18.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:18.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:18.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:19.507 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:19.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:19.507 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:19.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:19.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:19.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:19.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:19.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:19.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:19.228+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:19.229023+0000) 2024-02-20T09:13:19.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:19.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:19.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:19.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:19.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:19.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:19.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:19.323+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:19.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:19.505+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:19.506446+0000) 2024-02-20T09:13:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:19.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:19.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:19.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:19.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:19.990+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:19.990930+0000) 2024-02-20T09:13:20.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:20.230+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.231583+0000) 2024-02-20T09:13:20.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:20.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:20.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:20.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:20.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:20.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:20.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:20.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:20.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:20.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:20.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:20.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:20.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:20.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:20.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:20.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:20.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:20.508+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.508969+0000) 2024-02-20T09:13:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:20.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:20.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:20.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:20.921+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:20.990+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.991719+0000) 2024-02-20T09:13:21.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:21.231+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.232617+0000) 2024-02-20T09:13:21.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:21.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:21.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:21.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:21.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:21.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:21.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:21.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:21.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:21.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:21.334+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:21.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:21.335+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:21.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:21.509+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.509976+0000) 2024-02-20T09:13:21.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:21.689+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.690390+0000) 2024-02-20T09:13:21.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:21.688+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.689755+0000) 2024-02-20T09:13:21.812 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:21.692+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58740 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:21.812 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:21.690+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.690967+0000) 2024-02-20T09:13:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:21.990+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.991911+0000) 2024-02-20T09:13:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:21.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:21.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:21.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:21.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:22.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:22.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:22.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:22.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:22.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:22.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:22.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:22.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:22.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:22.231+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:22.232801+0000) 2024-02-20T09:13:22.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:22.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:22.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:22.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:22.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:22.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:22.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:22.366+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:22.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:22.509+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:22.510440+0000) 2024-02-20T09:13:23.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:22.994+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:22.995480+0000) 2024-02-20T09:13:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:22.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:22.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:22.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:49.591780+0000 (oldest deadline 2024-02-20T09:03:09.591780+0000) 2024-02-20T09:13:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:22.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:23.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:23.234+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:23.235460+0000) 2024-02-20T09:13:23.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:23.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:23.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:23.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:23.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:23.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:23.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:23.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:23.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:23.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:23.386+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:23.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:23.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:23.512+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:23.513129+0000) 2024-02-20T09:13:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:23.986+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:23.996+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:23.997884+0000) 2024-02-20T09:13:24.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:24.236+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:24.237926+0000) 2024-02-20T09:13:24.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:24.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:24.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:24.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:24.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:24.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:24.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:24.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:24.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:24.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:24.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:24.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:24.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:24.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:24.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:24.410+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:24.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:24.514+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:24.515593+0000) 2024-02-20T09:13:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:25.000+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:24.999+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.000514+0000) 2024-02-20T09:13:25.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:25.240+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.241448+0000) 2024-02-20T09:13:25.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:25.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:25.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:25.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:25.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:25.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:25.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:25.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:25.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:25.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:25.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:25.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:25.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:25.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:25.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:25.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:25.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:25.518+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.519037+0000) 2024-02-20T09:13:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:25.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:26.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:26.000+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.001390+0000) 2024-02-20T09:13:26.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:26.241+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.242484+0000) 2024-02-20T09:13:26.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:26.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:26.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:26.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:26.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:26.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:26.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:26.450+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:26.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:26.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:26.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:26.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:26.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:26.438+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:26.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:26.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:26.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:26.519+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.519973+0000) 2024-02-20T09:13:26.821 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:26.696+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.697460+0000) 2024-02-20T09:13:26.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:26.698+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58780 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:26.822 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:26.697+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.698238+0000) 2024-02-20T09:13:26.822 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:26.696+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.697633+0000) 2024-02-20T09:13:27.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:26.951+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:27.000+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:27.001578+0000) 2024-02-20T09:13:27.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:27.242+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:27.242968+0000) 2024-02-20T09:13:27.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:27.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:27.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:27.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:27.438+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:27.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:27.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:27.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:27.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:27.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:27.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:27.519+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:27.520164+0000) 2024-02-20T09:13:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:27.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:28.003+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:28.004030+0000) 2024-02-20T09:13:28.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:28.244+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:28.245354+0000) 2024-02-20T09:13:28.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:28.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:28.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:28.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:28.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:28.465+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:51.437668+0000 (oldest deadline 2024-02-20T09:03:11.437668+0000) 2024-02-20T09:13:28.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:28.465+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:28.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:28.521+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:28.522796+0000) 2024-02-20T09:13:28.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:28.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:28.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:28.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:28.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:28.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:28.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:28.426+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:28.877+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:29.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:29.006+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:29.007626+0000) 2024-02-20T09:13:29.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:29.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:29.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:29.247+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:29.248685+0000) 2024-02-20T09:13:29.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:29.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:29.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:29.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:29.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:29.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:29.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:29.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:29.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:29.524+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:29.525799+0000) 2024-02-20T09:13:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:29.928+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:30.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:30.010+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.011029+0000) 2024-02-20T09:13:30.529 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:30.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:30.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:30.250+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.251251+0000) 2024-02-20T09:13:30.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:30.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:30.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:30.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:30.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:30.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:30.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:30.476+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:30.838 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:30.527+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.528481+0000) 2024-02-20T09:13:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:30.972+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:31.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:31.011+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.012050+0000) 2024-02-20T09:13:31.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:31.250+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.251907+0000) 2024-02-20T09:13:31.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:31.501+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:31.827 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:31.528+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.529365+0000) 2024-02-20T09:13:31.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:31.703+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.703996+0000) 2024-02-20T09:13:31.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:31.706+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.707017+0000) 2024-02-20T09:13:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:31.706+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58819 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:31.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:31.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:31.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:31.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:31.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:31.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:31.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:31.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:31.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:31.704+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.705712+0000) 2024-02-20T09:13:32.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:31.923+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:32.095 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:32.011+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:32.012243+0000) 2024-02-20T09:13:32.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:32.251+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:32.252119+0000) 2024-02-20T09:13:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:32.522+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:32.528+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:32.529546+0000) 2024-02-20T09:13:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:32.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:32.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:32.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:32.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:32.552+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:32.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:33.013+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:33.014763+0000) 2024-02-20T09:13:33.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:33.253+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:33.254588+0000) 2024-02-20T09:13:33.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:33.530+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:33.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:33.530+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:33.531911+0000) 2024-02-20T09:13:33.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:33.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:33.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:33.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:33.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:33.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:02:54.544498+0000 (oldest deadline 2024-02-20T09:03:14.544498+0000) 2024-02-20T09:13:33.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:33.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:33.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:34.016+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:34.017066+0000) 2024-02-20T09:13:34.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:34.256+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:34.257084+0000) 2024-02-20T09:13:34.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:34.523+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:34.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:34.533+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:34.534772+0000) 2024-02-20T09:13:34.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:34.616+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:35.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:34.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:35.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:35.018+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.019384+0000) 2024-02-20T09:13:35.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:35.258+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.259673+0000) 2024-02-20T09:13:35.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:35.486+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:35.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:35.537+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.538320+0000) 2024-02-20T09:13:35.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:35.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:36.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:36.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:36.019+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.020268+0000) 2024-02-20T09:13:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:36.455+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:36.259+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.260754+0000) 2024-02-20T09:13:36.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:36.713+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58859 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:36.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:36.710+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.711678+0000) 2024-02-20T09:13:36.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:36.538+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.539356+0000) 2024-02-20T09:13:36.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:36.710+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.711925+0000) 2024-02-20T09:13:36.835 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:36.675+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:36.836 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:36.711+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.711961+0000) 2024-02-20T09:13:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:37.061+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:37.019+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:37.020742+0000) 2024-02-20T09:13:37.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:37.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:37.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:37.260952+0000) 2024-02-20T09:13:37.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:37.538+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:37.539595+0000) 2024-02-20T09:13:37.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:37.638+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:38.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:38.022+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:38.023012+0000) 2024-02-20T09:13:38.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:38.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:38.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:38.262+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:38.263314+0000) 2024-02-20T09:13:38.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:38.540+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:38.541694+0000) 2024-02-20T09:13:38.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:38.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:39.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:39.040+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:39.024+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:39.025576+0000) 2024-02-20T09:13:39.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:39.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:39.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:39.264+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:39.265705+0000) 2024-02-20T09:13:39.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:39.543+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:39.543971+0000) 2024-02-20T09:13:39.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:39.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:40.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:40.055+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:40.027+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.028359+0000) 2024-02-20T09:13:40.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:40.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:40.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:40.267+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.268282+0000) 2024-02-20T09:13:40.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:40.545+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.546572+0000) 2024-02-20T09:13:40.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:40.677+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:41.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:41.017+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:41.028+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.029464+0000) 2024-02-20T09:13:41.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:41.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:41.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:41.268+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.269239+0000) 2024-02-20T09:13:41.839 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:41.722+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58898 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:41.840 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:41.717+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.718894+0000) 2024-02-20T09:13:41.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:41.546+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.547636+0000) 2024-02-20T09:13:41.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:41.719+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.719933+0000) 2024-02-20T09:13:41.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:41.718+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.718962+0000) 2024-02-20T09:13:41.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:41.724+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:42.029+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:42.028+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:42.029632+0000) 2024-02-20T09:13:42.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:42.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:42.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:42.268+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:42.269408+0000) 2024-02-20T09:13:42.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:42.546+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:42.547835+0000) 2024-02-20T09:13:42.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:42.699+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:43.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:43.031+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:43.032701+0000) 2024-02-20T09:13:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:43.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:43.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:43.360+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:43.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:43.270+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:43.271795+0000) 2024-02-20T09:13:43.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:43.550+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:43.550933+0000) 2024-02-20T09:13:43.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:43.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:44.030+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:44.033+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:44.034910+0000) 2024-02-20T09:13:44.554 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:44.273+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:44.274481+0000) 2024-02-20T09:13:44.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:44.336+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:44.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:44.552+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:44.553643+0000) 2024-02-20T09:13:44.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:44.759+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:45.037+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.038198+0000) 2024-02-20T09:13:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:45.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:45.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:45.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:45.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:45.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:45.345+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:45.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:45.275+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.276615+0000) 2024-02-20T09:13:45.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:45.555+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.556191+0000) 2024-02-20T09:13:45.840 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:45.724+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:46.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:46.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:46.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:46.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:46.005+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:46.006+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:46.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:46.038+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.039345+0000) 2024-02-20T09:13:46.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:46.357+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:46.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:46.276+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.277680+0000) 2024-02-20T09:13:46.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:46.729+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58938 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:46.841 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:46.727+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.728748+0000) 2024-02-20T09:13:46.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:46.556+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.557037+0000) 2024-02-20T09:13:46.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:46.726+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.727720+0000) 2024-02-20T09:13:46.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:46.726+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.727547+0000) 2024-02-20T09:13:46.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:46.730+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:47.038+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:47.039528+0000) 2024-02-20T09:13:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:47.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:47.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:47.006+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:47.007+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:47.558 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:47.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:47.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:47.276+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:47.277926+0000) 2024-02-20T09:13:47.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:47.556+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:47.557226+0000) 2024-02-20T09:13:47.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:47.738+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:48.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:48.041+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:48.042823+0000) 2024-02-20T09:13:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.019+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.020+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:48.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:48.339+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:48.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.280+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:48.281375+0000) 2024-02-20T09:13:48.840 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:48.558+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:48.559736+0000) 2024-02-20T09:13:48.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:48.783+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:49.044+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:49.045126+0000) 2024-02-20T09:13:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:48.995+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:49.563 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:49.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:49.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:49.282+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:49.283459+0000) 2024-02-20T09:13:49.814 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:49.561+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:49.562101+0000) 2024-02-20T09:13:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:49.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:50.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:50.046+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.047692+0000) 2024-02-20T09:13:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:49.987+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:49.987+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:49.987+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:49.988+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:50.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:50.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:50.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:50.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:50.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:50.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:50.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:50.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:50.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:50.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.287018+0000) 2024-02-20T09:13:50.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:50.563+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.564699+0000) 2024-02-20T09:13:51.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:50.785+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:51.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:50.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:51.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:50.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:51.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:50.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:51.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:50.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:51.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:51.287+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.288145+0000) 2024-02-20T09:13:51.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:51.047+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.048427+0000) 2024-02-20T09:13:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:51.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:51.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:51.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:51.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:51.564+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.565496+0000) 2024-02-20T09:13:52.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:51.737+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 58977 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:52.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:51.734+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.735653+0000) 2024-02-20T09:13:52.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:51.735+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.736156+0000) 2024-02-20T09:13:52.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:51.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:52.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:51.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:52.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:51.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:52.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:51.972+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:52.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:51.734+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.734956+0000) 2024-02-20T09:13:52.051 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:51.833+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:52.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:52.047+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:52.048610+0000) 2024-02-20T09:13:52.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:52.287+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:52.288336+0000) 2024-02-20T09:13:52.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:52.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:52.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:52.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:52.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:52.564+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:52.565709+0000) 2024-02-20T09:13:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:52.816+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:53.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:53.051+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:53.052025+0000) 2024-02-20T09:13:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:52.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:52.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:52.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:52.943+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:53.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:53.290+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:53.291309+0000) 2024-02-20T09:13:53.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:53.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:53.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:53.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:53.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:53.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:53.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:53.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:53.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:53.566+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:53.567886+0000) 2024-02-20T09:13:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:53.856+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:54.054+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:54.055439+0000) 2024-02-20T09:13:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:53.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:53.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:53.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:53.914+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:54.571 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:54.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:54.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:54.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:54.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:54.426+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:54.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:54.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:54.572 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:54.292+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:54.293366+0000) 2024-02-20T09:13:54.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:54.569+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:54.570500+0000) 2024-02-20T09:13:54.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:54.813+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:55.056+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.058804+0000) 2024-02-20T09:13:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:54.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:54.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:54.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:54.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:55.574 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:55.293+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.295803+0000) 2024-02-20T09:13:55.574 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:55.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:55.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:55.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:55.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:55.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:55.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:55.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:55.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:55.571+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.573299+0000) 2024-02-20T09:13:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:55.847+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:56.058+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.059973+0000) 2024-02-20T09:13:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:55.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:55.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:55.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:55.923+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:56.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:56.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:56.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:56.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:56.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:56.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:56.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:56.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:56.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:56.294+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.296825+0000) 2024-02-20T09:13:56.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:13:56.743+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59017 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:13:56.842 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:56.740+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.742327+0000) 2024-02-20T09:13:56.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:56.741+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.743125+0000) 2024-02-20T09:13:56.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:56.572+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.574263+0000) 2024-02-20T09:13:56.843 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:56.740+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.742735+0000) 2024-02-20T09:13:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:56.891+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:57.058+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:57.060157+0000) 2024-02-20T09:13:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:56.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:56.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:56.941+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:57.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:56.942+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:57.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:57.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:57.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:57.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:57.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:57.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:57.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:57.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:57.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:57.295+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:57.297104+0000) 2024-02-20T09:13:57.841 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:57.572+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:57.574462+0000) 2024-02-20T09:13:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:57.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:57.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:57.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:57.937+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:57.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:58.060+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:58.062687+0000) 2024-02-20T09:13:58.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:58.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:58.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:58.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:58.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:58.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:58.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:58.290+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:58.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:58.297+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:58.299821+0000) 2024-02-20T09:13:58.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:58.576+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:58.577138+0000) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:58.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:58.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:58.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:58.933+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:59.064+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:59.065283+0000) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:58.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:58.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:58.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:13:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:58.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:59.580 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:59.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:59.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:59.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:59.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:59.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:13:59.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:59.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:13:59.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:59.301+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:59.302597+0000) 2024-02-20T09:13:59.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:13:59.578+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:59.579520+0000) 2024-02-20T09:14:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:59.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:59.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:59.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:00.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:13:59.975+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:59.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:59.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:59.962+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:13:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:13:59.963+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:00.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:00.066+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.067881+0000) 2024-02-20T09:14:00.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:00.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:00.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:00.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:00.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:00.291+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:00.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:00.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:00.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:00.304+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.305055+0000) 2024-02-20T09:14:00.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:00.582+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.582964+0000) 2024-02-20T09:14:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:00.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:00.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:00.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:00.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:00.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:00.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:00.977+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:00.978+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:01.067+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.068654+0000) 2024-02-20T09:14:01.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:01.305+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.306245+0000) 2024-02-20T09:14:01.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:01.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:01.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:01.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:01.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:01.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:01.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:01.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:01.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:01.582+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.583913+0000) 2024-02-20T09:14:01.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:01.752+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59056 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:01.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:01.749+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.750546+0000) 2024-02-20T09:14:01.843 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:01.749+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.750796+0000) 2024-02-20T09:14:01.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:01.750+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.751532+0000) 2024-02-20T09:14:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:01.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:01.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:01.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:01.938+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:01.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:01.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:01.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:01.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:02.067+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:02.068829+0000) 2024-02-20T09:14:02.585 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:02.305+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:02.306481+0000) 2024-02-20T09:14:02.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:02.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:02.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:02.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:02.585 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:02.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:02.586 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:02.263+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:02.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:02.583+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:02.584095+0000) 2024-02-20T09:14:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:02.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:02.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:02.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:02.980+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:02.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:02.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:02.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:02.935+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:03.070+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:03.071529+0000) 2024-02-20T09:14:03.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:03.307+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:03.308536+0000) 2024-02-20T09:14:03.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:03.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:03.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:03.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:03.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:03.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:03.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:03.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:03.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:03.585+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:03.586442+0000) 2024-02-20T09:14:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:03.973+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:03.973+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:03.973+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:03.974+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:03.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:03.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:03.970+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:03.970+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:04.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:04.073+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:04.074866+0000) 2024-02-20T09:14:04.590 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:04.311+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:04.312076+0000) 2024-02-20T09:14:04.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:04.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:04.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:04.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:04.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:04.266+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:04.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:04.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:04.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:04.588+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:04.589868+0000) 2024-02-20T09:14:05.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:05.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:05.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:05.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:05.008+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.008+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.009+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:05.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.076+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.077923+0000) 2024-02-20T09:14:05.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:05.313+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.314383+0000) 2024-02-20T09:14:05.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:05.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:05.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:05.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:05.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:05.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:05.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:05.279+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:05.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:05.591+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.592388+0000) 2024-02-20T09:14:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:06.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:06.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:06.056+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:06.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:05.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:06.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:06.077+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.078657+0000) 2024-02-20T09:14:06.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:06.314+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.315390+0000) 2024-02-20T09:14:06.594 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:06.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:06.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:06.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:06.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:06.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:06.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:06.309+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:06.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:06.759+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59096 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:06.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:06.755+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.756609+0000) 2024-02-20T09:14:06.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:06.756+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.757774+0000) 2024-02-20T09:14:06.880 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:06.592+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.593375+0000) 2024-02-20T09:14:06.880 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:06.757+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.758343+0000) 2024-02-20T09:14:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:07.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:07.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:07.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:07.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:07.045+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:06.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:06.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:07.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:06.984+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:06.985+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:07.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:07.077+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:07.078859+0000) 2024-02-20T09:14:07.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:07.314+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:07.315599+0000) 2024-02-20T09:14:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:07.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:07.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:07.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:07.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:07.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:07.842 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:07.592+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:07.593569+0000) 2024-02-20T09:14:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.000+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.001+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:07.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:07.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:07.963+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:07.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:08.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:08.080+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:08.081024+0000) 2024-02-20T09:14:08.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.318+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:08.319179+0000) 2024-02-20T09:14:08.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:08.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:08.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:08.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:08.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:08.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:08.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:08.256+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:08.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:08.594+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:08.595648+0000) 2024-02-20T09:14:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:08.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:08.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:08.932+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:08.933+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:09.082+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:09.083013+0000) 2024-02-20T09:14:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.970+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:08.971+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:09.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:09.320+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:09.321175+0000) 2024-02-20T09:14:09.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:09.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:09.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:09.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:09.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:09.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:09.599 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:09.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:09.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:09.596+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:09.597678+0000) 2024-02-20T09:14:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:09.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:09.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:09.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:09.999+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:09.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:09.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:09.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:09.937+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:10.084+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.084970+0000) 2024-02-20T09:14:10.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:10.322+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.323282+0000) 2024-02-20T09:14:10.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:10.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:10.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:10.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:10.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:10.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:10.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:10.268+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:10.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:10.600+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.601327+0000) 2024-02-20T09:14:11.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:10.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:11.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:10.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:10.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:10.960+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:10.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:10.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:10.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:10.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:11.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:11.085+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.086065+0000) 2024-02-20T09:14:11.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:11.323+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.323954+0000) 2024-02-20T09:14:11.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:11.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:11.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:11.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:11.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:11.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:11.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:11.258+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:11.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:11.768+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59135 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:11.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:11.765+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.765932+0000) 2024-02-20T09:14:11.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:11.601+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.602246+0000) 2024-02-20T09:14:11.889 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:11.764+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.765239+0000) 2024-02-20T09:14:11.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:11.764+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.765679+0000) 2024-02-20T09:14:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:11.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:11.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:11.969+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:11.969+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:11.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:11.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:11.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:11.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:12.085+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:12.086234+0000) 2024-02-20T09:14:12.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:12.323+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:12.324116+0000) 2024-02-20T09:14:12.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:12.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:12.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:12.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:12.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:12.255+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:12.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:12.256+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:12.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:12.601+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:12.602453+0000) 2024-02-20T09:14:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:12.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:12.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:12.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:12.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:12.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:12.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:12.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:12.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:13.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:13.088+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:13.089815+0000) 2024-02-20T09:14:13.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:13.326+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:13.327058+0000) 2024-02-20T09:14:13.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:13.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:13.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:13.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:13.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:13.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:13.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:13.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:13.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:13.604+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:13.605212+0000) 2024-02-20T09:14:14.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:13.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:14.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:13.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:14.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:13.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:14.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:13.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:14.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:13.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:14.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:13.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:14.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:13.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:14.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:13.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:14.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:14.091+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:14.092595+0000) 2024-02-20T09:14:14.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:14.329+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:14.330634+0000) 2024-02-20T09:14:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:14.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:14.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:14.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:14.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:14.886 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:14.607+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:14.608060+0000) 2024-02-20T09:14:15.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:14.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:15.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:14.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:15.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:14.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:15.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:14.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:15.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:14.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:15.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:14.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:15.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:14.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:15.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:14.886+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:15.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:15.095+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.095964+0000) 2024-02-20T09:14:15.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:15.332+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.333219+0000) 2024-02-20T09:14:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:15.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:15.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:15.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:15.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:15.862 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:15.609+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.610603+0000) 2024-02-20T09:14:15.862 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:15.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:15.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:15.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:15.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:15.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:15.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:15.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:16.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:15.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:15.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:15.864+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:15.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:16.096+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.097132+0000) 2024-02-20T09:14:16.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:16.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.334127+0000) 2024-02-20T09:14:16.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:16.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:16.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:16.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:16.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:16.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:16.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:16.228+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:16.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:16.777+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59171 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:16.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:16.772+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.773626+0000) 2024-02-20T09:14:16.877 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:16.610+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.611537+0000) 2024-02-20T09:14:16.877 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:16.772+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.773731+0000) 2024-02-20T09:14:16.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:16.772+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.773600+0000) 2024-02-20T09:14:16.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:16.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:16.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:16.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:16.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:16.892+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:16.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:16.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:16.875+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:17.096+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:17.097301+0000) 2024-02-20T09:14:17.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:17.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:17.334310+0000) 2024-02-20T09:14:17.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:17.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:17.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:17.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:17.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:17.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:17.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:17.270+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:17.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:17.610+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:17.611728+0000) 2024-02-20T09:14:17.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:17.826+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:17.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:17.826+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:17.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:17.826+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:17.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:17.827+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:18.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:18.099+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:18.100646+0000) 2024-02-20T09:14:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:17.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:17.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:17.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:18.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:17.895+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:18.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:18.335+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:18.336398+0000) 2024-02-20T09:14:18.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:18.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:18.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:18.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:18.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:18.221+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:18.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:18.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:18.613+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:18.614328+0000) 2024-02-20T09:14:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:18.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:18.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:18.856+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:18.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:18.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:18.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:18.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:18.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:19.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:19.102+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:19.103912+0000) 2024-02-20T09:14:19.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:19.337+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:19.338407+0000) 2024-02-20T09:14:19.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:19.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:19.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:19.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:19.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:19.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:19.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:19.223+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:19.616+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:19.617847+0000) 2024-02-20T09:14:20.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:19.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:20.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:19.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:20.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:19.835+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:20.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:19.836+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:20.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:19.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:20.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:19.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:20.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:19.794+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:20.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:19.795+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:20.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:20.339+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.340587+0000) 2024-02-20T09:14:20.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:20.105+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.106388+0000) 2024-02-20T09:14:20.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:20.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:20.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:20.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:20.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:20.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:20.619+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.620423+0000) 2024-02-20T09:14:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:20.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:20.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:20.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:20.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:20.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:20.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:20.776+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:20.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:20.777+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:21.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:21.340+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.341758+0000) 2024-02-20T09:14:21.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:21.106+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.107563+0000) 2024-02-20T09:14:21.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:21.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:21.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:21.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:21.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:21.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:21.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:21.235+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:21.907 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:21.785+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59210 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:21.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:21.620+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.621506+0000) 2024-02-20T09:14:21.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:21.782+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.783287+0000) 2024-02-20T09:14:21.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:21.781+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.782803+0000) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:21.833+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:21.833+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:21.833+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:21.834+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:21.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:21.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:21.908 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:21.748+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:21.909 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:21.749+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:21.909 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:21.781+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.782582+0000) 2024-02-20T09:14:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:22.106+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:22.107751+0000) 2024-02-20T09:14:22.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:22.341+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:22.341953+0000) 2024-02-20T09:14:22.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:22.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:22.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:22.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:22.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:22.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:22.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:22.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:22.620+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:22.621727+0000) 2024-02-20T09:14:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:22.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:22.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:22.727+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:22.728+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:22.804+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:22.804+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:22.804+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:22.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:22.805+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:23.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:23.344+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:23.345251+0000) 2024-02-20T09:14:23.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:23.110+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:23.111011+0000) 2024-02-20T09:14:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:23.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:23.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:23.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:23.263+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:23.624+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:23.624968+0000) 2024-02-20T09:14:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:23.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:23.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:23.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:23.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:23.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:23.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:23.733+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:23.733+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:24.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:24.347+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:24.348590+0000) 2024-02-20T09:14:24.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:24.112+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:24.113564+0000) 2024-02-20T09:14:24.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:24.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:24.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:24.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:24.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:24.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:24.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:24.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:24.626+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:24.627570+0000) 2024-02-20T09:14:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:24.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:24.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:24.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:24.790+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:24.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:24.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:24.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:24.723+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:25.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:25.351+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.352178+0000) 2024-02-20T09:14:25.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:25.115+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.116184+0000) 2024-02-20T09:14:25.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:25.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:25.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:25.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:25.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:25.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:25.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:25.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:25.630+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.631101+0000) 2024-02-20T09:14:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:25.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:25.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:25.791+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:25.792+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:25.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:25.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:25.760+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:25.761+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:26.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:26.352+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.353339+0000) 2024-02-20T09:14:26.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:26.116+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.116971+0000) 2024-02-20T09:14:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:26.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:26.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:26.192+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:26.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:26.193+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:26.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:26.793+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59250 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:26.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:26.631+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.632237+0000) 2024-02-20T09:14:26.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:26.790+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.791767+0000) 2024-02-20T09:14:26.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:26.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:26.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:26.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:26.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:26.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:26.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:26.789+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:26.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:26.790+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.791840+0000) 2024-02-20T09:14:26.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:26.790+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.791203+0000) 2024-02-20T09:14:26.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:26.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:26.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:26.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:26.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:26.795+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:26.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:26.796+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:27.116+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:27.117157+0000) 2024-02-20T09:14:27.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:27.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:27.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:27.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:27.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:27.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:27.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:27.352+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:27.353530+0000) 2024-02-20T09:14:27.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:27.631+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:27.632408+0000) 2024-02-20T09:14:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:27.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:27.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:27.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:27.779+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:27.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:27.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:27.814+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:27.815+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:28.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:28.356+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:28.357110+0000) 2024-02-20T09:14:28.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:28.118+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:28.119771+0000) 2024-02-20T09:14:28.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:28.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:28.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:28.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:28.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:28.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:28.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:28.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:28.633+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:28.634911+0000) 2024-02-20T09:14:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:28.823+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:28.823+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:28.823+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:28.823+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:28.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:28.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:28.732+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:28.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:28.733+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:29.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:29.359+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:29.359947+0000) 2024-02-20T09:14:29.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:29.121+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:29.122289+0000) 2024-02-20T09:14:29.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:29.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:29.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:29.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:29.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:29.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:29.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:29.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:29.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:29.636+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:29.637294+0000) 2024-02-20T09:14:29.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:29.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:29.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:29.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:29.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:29.697+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:29.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:29.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:29.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:29.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:29.857+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:30.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:30.361+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.362183+0000) 2024-02-20T09:14:30.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:30.123+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.124743+0000) 2024-02-20T09:14:30.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:30.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:30.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:30.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:30.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:30.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:30.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:30.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:30.639+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.640862+0000) 2024-02-20T09:14:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:30.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:30.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:30.662+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:30.663+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:30.848+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:30.848+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:30.848+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:30.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:30.850+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:31.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:31.124+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.125701+0000) 2024-02-20T09:14:31.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:31.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:31.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:31.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:31.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:31.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:31.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:31.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:31.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:31.362+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.362929+0000) 2024-02-20T09:14:31.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:31.641+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.641951+0000) 2024-02-20T09:14:31.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:31.798+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.799537+0000) 2024-02-20T09:14:31.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:31.641+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:31.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:31.641+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:31.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:31.641+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:31.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:31.642+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:31.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:31.798+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.799560+0000) 2024-02-20T09:14:31.922 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:31.800+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59290 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:31.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:31.797+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.798410+0000) 2024-02-20T09:14:31.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:31.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:31.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:31.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:31.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:31.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:31.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:31.822+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:32.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:32.124+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:32.125900+0000) 2024-02-20T09:14:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:32.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:32.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:32.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:32.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:32.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:32.362+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:32.363116+0000) 2024-02-20T09:14:32.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:32.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:32.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:32.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:32.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:32.632+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:32.641+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:32.642137+0000) 2024-02-20T09:14:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:32.633+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:32.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:32.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:32.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:32.831+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:33.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:33.128+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:33.129344+0000) 2024-02-20T09:14:33.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:33.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:33.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:33.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:33.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:33.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:33.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:33.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:33.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:33.365+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:33.366498+0000) 2024-02-20T09:14:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:33.643+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:33.644268+0000) 2024-02-20T09:14:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:33.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:33.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:33.656+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:33.656+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:33.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:33.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:33.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:33.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:33.800+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:34.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:34.130+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:34.131359+0000) 2024-02-20T09:14:34.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:34.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:34.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:34.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:34.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:34.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:34.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:34.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:34.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:34.369+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:34.369952+0000) 2024-02-20T09:14:34.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:34.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:34.645+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:34.646799+0000) 2024-02-20T09:14:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:34.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:34.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:34.608+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:34.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:34.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:34.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:34.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:34.844+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:35.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:35.132+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.133402+0000) 2024-02-20T09:14:35.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:35.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:35.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:35.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:35.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:35.371+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.372392+0000) 2024-02-20T09:14:35.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:35.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:35.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:35.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:35.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:35.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:35.845 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.649+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.650260+0000) 2024-02-20T09:14:35.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:35.587+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:36.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:35.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:36.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:35.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:36.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:35.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:36.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:35.883+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:36.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:36.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:36.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.983+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:36.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:35.984+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:36.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:36.372+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.373191+0000) 2024-02-20T09:14:36.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:36.133+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.134335+0000) 2024-02-20T09:14:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:36.650+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.651516+0000) 2024-02-20T09:14:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:36.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:36.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:36.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:36.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:37.135 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:36.807+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59330 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:37.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:36.806+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.807119+0000) 2024-02-20T09:14:37.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:36.805+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.806273+0000) 2024-02-20T09:14:37.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:36.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:37.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:36.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:37.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:36.999+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:37.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:37.000+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:37.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:36.804+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.805367+0000) 2024-02-20T09:14:37.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:36.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:37.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:36.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:37.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:36.859+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:37.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:36.860+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:37.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:37.133+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:37.134519+0000) 2024-02-20T09:14:37.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:37.372+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:37.373382+0000) 2024-02-20T09:14:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:37.650+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:37.651728+0000) 2024-02-20T09:14:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:37.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:37.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:37.618+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:37.619+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:38.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:38.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:38.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:38.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:38.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:38.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:38.138 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:38.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:38.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:37.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:38.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:37.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:38.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:37.844+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:38.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:37.845+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:38.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:38.375+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:38.376893+0000) 2024-02-20T09:14:38.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:38.135+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:38.136771+0000) 2024-02-20T09:14:38.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:38.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:38.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:38.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:38.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:38.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:38.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:38.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:38.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:38.653+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:38.653974+0000) 2024-02-20T09:14:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:38.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:38.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:38.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:38.802+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:39.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:39.140+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:39.141302+0000) 2024-02-20T09:14:39.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:39.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:39.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:39.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:39.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:39.077+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:39.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:39.078+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:39.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:39.378+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:39.379488+0000) 2024-02-20T09:14:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:39.656+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:39.656930+0000) 2024-02-20T09:14:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:39.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:39.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:39.607+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:39.608+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:39.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:39.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:39.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:39.851+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:40.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:40.143+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.144015+0000) 2024-02-20T09:14:40.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:40.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:40.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:40.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:40.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:40.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:40.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:40.045+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:40.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:40.382+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.383088+0000) 2024-02-20T09:14:40.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:40.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:40.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:40.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:40.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:40.612+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:40.658+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.659065+0000) 2024-02-20T09:14:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:40.613+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:40.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:40.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:40.850+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:40.851+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:41.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:41.143+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.144831+0000) 2024-02-20T09:14:41.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:41.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:41.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:41.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:41.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:41.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:41.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:41.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:41.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:41.383+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.384050+0000) 2024-02-20T09:14:41.921 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:41.815+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59369 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:41.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:41.659+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.660209+0000) 2024-02-20T09:14:41.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:41.811+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.812793+0000) 2024-02-20T09:14:41.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:41.812+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.813590+0000) 2024-02-20T09:14:41.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:41.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:41.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:41.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:41.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:41.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:41.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:41.836+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:41.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:41.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:41.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:41.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:41.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:41.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:41.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:41.621+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:41.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:41.812+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.813279+0000) 2024-02-20T09:14:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:42.144+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:42.145034+0000) 2024-02-20T09:14:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:42.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:42.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:42.036+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:42.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:42.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:42.383+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:42.384231+0000) 2024-02-20T09:14:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:42.659+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:42.660456+0000) 2024-02-20T09:14:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:42.620+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:42.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:42.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:42.621+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:42.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:42.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:42.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:42.831+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:43.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:43.147+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:43.148020+0000) 2024-02-20T09:14:43.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:43.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:43.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:43.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:43.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:43.064+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:43.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:43.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:43.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:43.386+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:43.387815+0000) 2024-02-20T09:14:43.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:43.661+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:43.662888+0000) 2024-02-20T09:14:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:43.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:43.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:43.878+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:43.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:43.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:43.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:43.634+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:43.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:43.635+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:44.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:44.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:44.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:44.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:44.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:44.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:44.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:44.039+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:44.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:44.150+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:44.151588+0000) 2024-02-20T09:14:44.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:44.389+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:44.390120+0000) 2024-02-20T09:14:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:44.664+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:44.665540+0000) 2024-02-20T09:14:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:44.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:44.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:44.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:44.623+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:44.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:44.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:44.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:44.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:44.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:44.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:45.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:45.153+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.154127+0000) 2024-02-20T09:14:45.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:45.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:45.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:45.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:45.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:45.046+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:45.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:45.047+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:45.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:45.392+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.392984+0000) 2024-02-20T09:14:45.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:45.667+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.668110+0000) 2024-02-20T09:14:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:45.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:45.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:45.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:45.859+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:45.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:45.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:45.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:45.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:45.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:45.661+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:46.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:46.154+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.154932+0000) 2024-02-20T09:14:46.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:46.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:46.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:46.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:46.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:46.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:46.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:46.064+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:46.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:46.393+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.394188+0000) 2024-02-20T09:14:46.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:46.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:46.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:46.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:46.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:46.639+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:46.945 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:46.822+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59409 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:46.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:46.668+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.669416+0000) 2024-02-20T09:14:46.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:46.823+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.824171+0000) 2024-02-20T09:14:46.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:46.639+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:46.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:46.820+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.821167+0000) 2024-02-20T09:14:46.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:46.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:46.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:46.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:46.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:46.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:46.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:46.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:46.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:46.820+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.821069+0000) 2024-02-20T09:14:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:47.154+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:47.155104+0000) 2024-02-20T09:14:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:47.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:47.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:47.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:47.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:47.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:47.393+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:47.394360+0000) 2024-02-20T09:14:47.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:47.668+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:47.669600+0000) 2024-02-20T09:14:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:47.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:47.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:47.689+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:47.690+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:47.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:47.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:47.841+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:47.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:47.841+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:48.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:48.156+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:48.157793+0000) 2024-02-20T09:14:48.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:48.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:48.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:48.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:48.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:48.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:48.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:48.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:48.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:48.395+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:48.397276+0000) 2024-02-20T09:14:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:48.670+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:48.672201+0000) 2024-02-20T09:14:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:48.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:48.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:48.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:48.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:48.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:48.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:48.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:48.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:48.877+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:49.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:49.159+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:49.161129+0000) 2024-02-20T09:14:49.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:49.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:49.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:49.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:49.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:49.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:49.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:49.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:49.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:49.398+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:49.400732+0000) 2024-02-20T09:14:49.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:49.672+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:49.674814+0000) 2024-02-20T09:14:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:49.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:49.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:49.676+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:49.677+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:49.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:49.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:49.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:49.839+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:50.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:50.162+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.164560+0000) 2024-02-20T09:14:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:50.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:50.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:50.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:50.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:50.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:50.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:50.402+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.404051+0000) 2024-02-20T09:14:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:50.676+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.678443+0000) 2024-02-20T09:14:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:50.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:50.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:50.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:50.839+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:50.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:50.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:50.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:50.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:51.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:51.163+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.165781+0000) 2024-02-20T09:14:51.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:51.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:51.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:51.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:51.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:51.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:51.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:51.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:51.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:51.402+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.404922+0000) 2024-02-20T09:14:51.952 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:51.830+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59448 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:51.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:51.678+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.679684+0000) 2024-02-20T09:14:51.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:51.827+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.828414+0000) 2024-02-20T09:14:51.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:51.827+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.828149+0000) 2024-02-20T09:14:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:51.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:51.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:51.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:51.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:51.885+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:51.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:51.766+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:51.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:51.766+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:51.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:51.766+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:51.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:51.767+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:51.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:51.826+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.827642+0000) 2024-02-20T09:14:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:52.165+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:52.166028+0000) 2024-02-20T09:14:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:52.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:52.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:52.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:52.050+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:52.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:52.404+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:52.405130+0000) 2024-02-20T09:14:52.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:52.678+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:52.679880+0000) 2024-02-20T09:14:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:52.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:52.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:52.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:52.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:52.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:52.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:52.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:52.904+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:53.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:53.167+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:53.168454+0000) 2024-02-20T09:14:53.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:53.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:53.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:53.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:53.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:53.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:53.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:53.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:53.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:53.407+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:53.408697+0000) 2024-02-20T09:14:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:53.681+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:53.681949+0000) 2024-02-20T09:14:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:53.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:53.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:53.784+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:53.785+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:53.901+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:53.901+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:53.901+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:53.902+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:54.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:54.169+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:54.170907+0000) 2024-02-20T09:14:54.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:54.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:54.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:54.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:54.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:54.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:54.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:54.410+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:54.411261+0000) 2024-02-20T09:14:54.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:54.684+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:54.685361+0000) 2024-02-20T09:14:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:54.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:54.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:54.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:54.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:54.880+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:54.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:54.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:54.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:54.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:54.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:54.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:55.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:55.172+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.173208+0000) 2024-02-20T09:14:55.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:55.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:55.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:55.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:55.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:55.042+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:55.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:55.043+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:55.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:55.413+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.414737+0000) 2024-02-20T09:14:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:55.687+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.688887+0000) 2024-02-20T09:14:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:55.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:55.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:55.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:55.787+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:55.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:55.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:55.867+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:55.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:56.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:56.173+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.174190+0000) 2024-02-20T09:14:56.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:56.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:56.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.007+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:56.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.007+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:56.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:56.414+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.415918+0000) 2024-02-20T09:14:56.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:14:56.839+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59488 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:14:56.958 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.688+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.689898+0000) 2024-02-20T09:14:56.958 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.834+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.835718+0000) 2024-02-20T09:14:56.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:56.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:56.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:56.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:56.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:56.749+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:56.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:56.750+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:56.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:56.835+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.836437+0000) 2024-02-20T09:14:56.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:56.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:56.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:56.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:56.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:56.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:56.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:56.831+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:56.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:56.835+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.836119+0000) 2024-02-20T09:14:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:57.173+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:57.174382+0000) 2024-02-20T09:14:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.980+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.980+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.980+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:56.981+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:57.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:57.415+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:57.416123+0000) 2024-02-20T09:14:57.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:57.689+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:57.690112+0000) 2024-02-20T09:14:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:57.783+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:57.783+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:57.783+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:57.784+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:57.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:57.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:57.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:57.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:58.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:58.176+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:58.177681+0000) 2024-02-20T09:14:58.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:57.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:58.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:57.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:58.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:57.981+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:58.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:57.982+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:58.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:58.417+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:58.418187+0000) 2024-02-20T09:14:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:58.691+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:58.692693+0000) 2024-02-20T09:14:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:58.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:58.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:58.748+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:58.749+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:58.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:58.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:58.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:14:58.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:58.927+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:59.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:59.179+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:59.179954+0000) 2024-02-20T09:14:59.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:59.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:59.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:59.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:59.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:59.029+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:14:59.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:59.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:14:59.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:59.419+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:59.420698+0000) 2024-02-20T09:14:59.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:14:59.693+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:59.694720+0000) 2024-02-20T09:14:59.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:59.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:59.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:59.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:59.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:59.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:14:59.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:14:59.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:00.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:00.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:00.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:00.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:59.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:59.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:59.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:14:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:14:59.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:00.181+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.182654+0000) 2024-02-20T09:15:00.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:00.422+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.423302+0000) 2024-02-20T09:15:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:00.697+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.698247+0000) 2024-02-20T09:15:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:00.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:00.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:00.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:00.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:01.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:01.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:01.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:01.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:01.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:01.015+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:01.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:01.016+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:01.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:00.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:01.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:00.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:01.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:00.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:01.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:00.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:01.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:01.183+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.183959+0000) 2024-02-20T09:15:01.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:01.423+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.424379+0000) 2024-02-20T09:15:01.966 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:01.846+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59527 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:01.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:01.698+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.699653+0000) 2024-02-20T09:15:01.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:01.843+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.844173+0000) 2024-02-20T09:15:01.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:01.842+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.843909+0000) 2024-02-20T09:15:01.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:01.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:01.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:01.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:01.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:01.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:01.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:01.725+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:01.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:01.843+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.844472+0000) 2024-02-20T09:15:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:02.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:02.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:02.048+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:02.049+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.183+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:02.184132+0000) 2024-02-20T09:15:02.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:02.423+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:02.424539+0000) 2024-02-20T09:15:02.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:02.699+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:02.699932+0000) 2024-02-20T09:15:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:02.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:02.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:02.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:02.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:03.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:03.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:03.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:03.070+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.958+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:02.959+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:03.185+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:03.186235+0000) 2024-02-20T09:15:03.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:03.426+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:03.427794+0000) 2024-02-20T09:15:03.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:03.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:03.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:03.702+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:03.703419+0000) 2024-02-20T09:15:03.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:03.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:03.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:03.677+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:03.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:03.678+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:03.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:03.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:03.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:03.961+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:04.187+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:04.188429+0000) 2024-02-20T09:15:04.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:04.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:04.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:04.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:04.055+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:04.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:04.430+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:04.431416+0000) 2024-02-20T09:15:04.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:04.705+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:04.706060+0000) 2024-02-20T09:15:04.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:04.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:04.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:04.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:04.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:04.683+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:04.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:04.684+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:05.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:05.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:05.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:05.026+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:04.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:04.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:04.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:04.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:05.191+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.192066+0000) 2024-02-20T09:15:05.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:05.432+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.433593+0000) 2024-02-20T09:15:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:05.707+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.708141+0000) 2024-02-20T09:15:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:05.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:05.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:05.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:05.711+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:05.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:05.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:05.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:05.933+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:06.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:06.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:06.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:06.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:06.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:06.066+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:06.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:06.067+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:06.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:06.192+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.193441+0000) 2024-02-20T09:15:06.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:06.434+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.434973+0000) 2024-02-20T09:15:06.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:06.854+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59567 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:06.851+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.852838+0000) 2024-02-20T09:15:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:06.708+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.709560+0000) 2024-02-20T09:15:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:06.851+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.852767+0000) 2024-02-20T09:15:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:06.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:06.705+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:06.706+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:06.707+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:06.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:06.850+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.851393+0000) 2024-02-20T09:15:07.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:07.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.036+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:06.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:06.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:06.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:06.972+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:07.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:07.192+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:07.193641+0000) 2024-02-20T09:15:07.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:07.434+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:07.435168+0000) 2024-02-20T09:15:07.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:07.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:07.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:07.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:07.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:07.661+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:07.933 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.708+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:07.709769+0000) 2024-02-20T09:15:07.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:07.662+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:08.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:08.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:08.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:08.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:07.989+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:08.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:07.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:08.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:07.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:08.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:07.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:08.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:07.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:08.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:08.436+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:08.437216+0000) 2024-02-20T09:15:08.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:08.194+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:08.195883+0000) 2024-02-20T09:15:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:08.710+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:08.711761+0000) 2024-02-20T09:15:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:08.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:08.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:08.951+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:08.952+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:08.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:08.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:08.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:08.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:08.703+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:09.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:08.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:09.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:08.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:09.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:08.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:09.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:08.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:09.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:09.196+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:09.197902+0000) 2024-02-20T09:15:09.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:09.440+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:09.440985+0000) 2024-02-20T09:15:09.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:09.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:09.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:09.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:09.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:09.666+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:09.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:09.668+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:09.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:09.713+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:09.713957+0000) 2024-02-20T09:15:09.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:09.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:09.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:09.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:09.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:09.957+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:09.957+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:10.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:10.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:10.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:10.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:10.200+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.201373+0000) 2024-02-20T09:15:10.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:10.444+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.445123+0000) 2024-02-20T09:15:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:10.715+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.716565+0000) 2024-02-20T09:15:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:10.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:10.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:10.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:10.701+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:11.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:10.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:11.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:10.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:11.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:10.992+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:11.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:10.992+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:11.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:11.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:11.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:11.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:11.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:11.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:11.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:11.046+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:11.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:11.201+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.202426+0000) 2024-02-20T09:15:11.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:11.445+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.446070+0000) 2024-02-20T09:15:11.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:11.862+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59606 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:11.859+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.860692+0000) 2024-02-20T09:15:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:11.716+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.717425+0000) 2024-02-20T09:15:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:11.858+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.859612+0000) 2024-02-20T09:15:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:11.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:11.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:11.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:11.721+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:11.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:11.859+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.860513+0000) 2024-02-20T09:15:12.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:11.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:12.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:11.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:12.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:11.995+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:12.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:11.996+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:12.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:12.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:12.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:12.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:12.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:12.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:12.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:12.073+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:12.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:12.201+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:12.202607+0000) 2024-02-20T09:15:12.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:12.445+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:12.446253+0000) 2024-02-20T09:15:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:12.716+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:12.717631+0000) 2024-02-20T09:15:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:12.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:12.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:12.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:12.769+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:13.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:13.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:13.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:13.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:13.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:13.012+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:13.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:13.013+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:13.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:13.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:13.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:13.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:13.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.205+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:13.206101+0000) 2024-02-20T09:15:13.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:13.447+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:13.448554+0000) 2024-02-20T09:15:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:13.719+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:13.720472+0000) 2024-02-20T09:15:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:13.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:13.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:13.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:13.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:14.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:14.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:14.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:14.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:14.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:14.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:14.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:14.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:14.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:14.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:14.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.999+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:14.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:13.999+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:14.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:14.208+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:14.209523+0000) 2024-02-20T09:15:14.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:14.450+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:14.451015+0000) 2024-02-20T09:15:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:14.721+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:14.722737+0000) 2024-02-20T09:15:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:14.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:14.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:14.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:14.727+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:15.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:15.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:15.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:15.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:15.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:15.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:15.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:15.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:15.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:14.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:15.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:14.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:15.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:14.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:15.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:14.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:15.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:15.212+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:15.213102+0000) 2024-02-20T09:15:15.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:15.453+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:15.454639+0000) 2024-02-20T09:15:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:15.724+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:15.725076+0000) 2024-02-20T09:15:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:15.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:15.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:15.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:15.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:16.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:16.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:16.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:16.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:16.277 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:16.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:16.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:16.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:16.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:16.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:16.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:16.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:16.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:16.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:16.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:16.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:16.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:16.213+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.213971+0000) 2024-02-20T09:15:16.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:16.455+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.456061+0000) 2024-02-20T09:15:16.867 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:16.725+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.726291+0000) 2024-02-20T09:15:16.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:16.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:16.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:16.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:16.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:16.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:16.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:16.765+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:17.178 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:16.870+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59646 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:17.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:16.866+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.867194+0000) 2024-02-20T09:15:17.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:16.869+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.870258+0000) 2024-02-20T09:15:17.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:16.866+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.867505+0000) 2024-02-20T09:15:17.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:17.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:17.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:17.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:17.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:17.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:17.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:17.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:17.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:17.213+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:17.214147+0000) 2024-02-20T09:15:17.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:17.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:17.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:17.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:17.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:17.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:17.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:17.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:17.455+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:17.456233+0000) 2024-02-20T09:15:18.074 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:17.725+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:17.726487+0000) 2024-02-20T09:15:18.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:17.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:18.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:17.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:18.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:17.738+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:18.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:17.738+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:18.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:18.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:18.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:18.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:18.075 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:18.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:18.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:18.073+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:18.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:18.215+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:18.216506+0000) 2024-02-20T09:15:18.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:18.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:18.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:18.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:18.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:18.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:18.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:18.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:18.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:18.457+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:18.458617+0000) 2024-02-20T09:15:18.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:18.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:18.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:18.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:18.728+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:18.729744+0000) 2024-02-20T09:15:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:18.708+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:18.709+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:19.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:19.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:19.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:19.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:19.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:19.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:19.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:19.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:19.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:19.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:19.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:19.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:19.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:19.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:19.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:19.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:19.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:19.218+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:19.219079+0000) 2024-02-20T09:15:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:19.459+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:19.460547+0000) 2024-02-20T09:15:20.148 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:19.732+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:19.733311+0000) 2024-02-20T09:15:20.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:19.730+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:20.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:19.730+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:20.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:19.730+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:20.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:19.731+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:20.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:20.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:20.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:20.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:20.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:20.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:20.409 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:20.150+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:20.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:20.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:20.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:20.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:20.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:20.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:20.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:20.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:20.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:20.220+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:20.221270+0000) 2024-02-20T09:15:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:20.462+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:20.463049+0000) 2024-02-20T09:15:21.112 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:20.734+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:20.735478+0000) 2024-02-20T09:15:21.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:20.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:21.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:20.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:21.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:20.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:21.113 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:20.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:21.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:21.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:21.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:21.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:21.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:21.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:21.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:21.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:21.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:21.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:21.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:21.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:21.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:21.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:21.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:21.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:21.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:21.221+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.222153+0000) 2024-02-20T09:15:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:21.463+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.464425+0000) 2024-02-20T09:15:21.997 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:21.877+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59685 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:21.997 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:21.875+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.876766+0000) 2024-02-20T09:15:21.998 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:21.735+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.736332+0000) 2024-02-20T09:15:21.998 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:21.874+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.875448+0000) 2024-02-20T09:15:21.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:21.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:21.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:21.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:21.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:21.793+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:21.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:21.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:21.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:21.874+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.875364+0000) 2024-02-20T09:15:22.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:22.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:22.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:22.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:22.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:22.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:22.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:22.200+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:22.278 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:22.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:22.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:22.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:22.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:22.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:22.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:22.221+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:22.222337+0000) 2024-02-20T09:15:22.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:22.463+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:22.464609+0000) 2024-02-20T09:15:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:22.735+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:22.736537+0000) 2024-02-20T09:15:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:22.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:22.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:22.764+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:22.764+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:23.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:23.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:23.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:23.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:23.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:23.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:23.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:23.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:23.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:23.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:23.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:23.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:23.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:23.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:23.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:23.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:23.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:23.223+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:23.224728+0000) 2024-02-20T09:15:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:23.465+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:23.466747+0000) 2024-02-20T09:15:24.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:23.737+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:23.738748+0000) 2024-02-20T09:15:24.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:23.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:24.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:23.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:24.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:23.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:24.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:23.781+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:24.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:24.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:24.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:24.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:24.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:24.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:24.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:24.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:24.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:24.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:24.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:24.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:24.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:24.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:24.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:24.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:24.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:24.227+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:24.228162+0000) 2024-02-20T09:15:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:24.468+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:24.469373+0000) 2024-02-20T09:15:25.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:24.740+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:24.741219+0000) 2024-02-20T09:15:25.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:24.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:25.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:24.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:25.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:24.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:25.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:24.768+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:25.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:25.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:25.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:25.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:25.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:25.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:25.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:25.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:25.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:25.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:25.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:25.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:25.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:25.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:25.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:25.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:25.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:25.229+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:25.230420+0000) 2024-02-20T09:15:25.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:25.471+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:25.472762+0000) 2024-02-20T09:15:26.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:25.743+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:25.744660+0000) 2024-02-20T09:15:26.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:26.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:26.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:26.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:26.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:25.739+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:26.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:25.739+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:26.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:25.739+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:26.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:25.740+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:26.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:26.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:26.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:26.125+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:26.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:26.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:26.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:26.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:26.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:26.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:26.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:26.157+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:26.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:26.230+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.231315+0000) 2024-02-20T09:15:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:26.473+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.474133+0000) 2024-02-20T09:15:27.005 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:26.885+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59725 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:27.005 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:26.744+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.745883+0000) 2024-02-20T09:15:27.005 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:26.883+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.884185+0000) 2024-02-20T09:15:27.006 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:26.884+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.885223+0000) 2024-02-20T09:15:27.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:26.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:27.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:26.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:27.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:26.744+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:27.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:26.745+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:27.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:26.882+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.883384+0000) 2024-02-20T09:15:27.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:27.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:27.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:27.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:27.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:27.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:27.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:27.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:27.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:27.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:27.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:27.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:27.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:27.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:27.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:27.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:27.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:27.230+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:27.231500+0000) 2024-02-20T09:15:27.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:27.473+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:27.474337+0000) 2024-02-20T09:15:27.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:27.745+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:27.746114+0000) 2024-02-20T09:15:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:27.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:27.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:27.767+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:27.768+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:28.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:28.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:28.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:28.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:28.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:28.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:28.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:28.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:28.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:28.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:28.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:28.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:28.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:28.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:28.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:28.095+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:28.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:28.232+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:28.233727+0000) 2024-02-20T09:15:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:28.475+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:28.476898+0000) 2024-02-20T09:15:29.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:28.747+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:28.748188+0000) 2024-02-20T09:15:29.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:28.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:29.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:28.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:29.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:28.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:29.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:28.742+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:29.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:29.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:29.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:29.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:29.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:29.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:29.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:29.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:29.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:29.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:29.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:29.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:29.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:29.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:29.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:29.069+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:29.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:29.235+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:29.235956+0000) 2024-02-20T09:15:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:29.478+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:29.479086+0000) 2024-02-20T09:15:30.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:29.750+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:29.751728+0000) 2024-02-20T09:15:30.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:29.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:30.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:29.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:30.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:29.747+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:30.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:29.748+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:30.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:30.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:30.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:30.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:30.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:30.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:30.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:30.095+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:30.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:30.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:30.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:30.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:30.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:30.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:30.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:30.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:30.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:30.237+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:30.238170+0000) 2024-02-20T09:15:30.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:30.480+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:30.481750+0000) 2024-02-20T09:15:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:30.754+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:30.755312+0000) 2024-02-20T09:15:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:30.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:30.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:30.756+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:30.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:31.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:31.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:31.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:31.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:31.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:31.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:31.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:31.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:31.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:31.238+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.239555+0000) 2024-02-20T09:15:31.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:31.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:31.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:31.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:31.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:31.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:31.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:31.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:31.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:31.481+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.482712+0000) 2024-02-20T09:15:31.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:31.892+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59763 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:31.889+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.890533+0000) 2024-02-20T09:15:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:31.755+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.756473+0000) 2024-02-20T09:15:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:31.889+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.890916+0000) 2024-02-20T09:15:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:31.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:31.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:31.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:31.713+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:31.889+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.890635+0000) 2024-02-20T09:15:32.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:32.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:32.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:32.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:32.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:32.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:32.279 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:32.127+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:32.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:32.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:32.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:32.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:32.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:32.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:32.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:32.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:32.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:32.238+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:32.239756+0000) 2024-02-20T09:15:32.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:32.481+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:32.482903+0000) 2024-02-20T09:15:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:32.755+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:32.756635+0000) 2024-02-20T09:15:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:32.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:32.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:32.665+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:32.666+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:33.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:33.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:33.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:33.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:33.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:33.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:33.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:33.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:33.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:33.241+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:33.241955+0000) 2024-02-20T09:15:33.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:33.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:33.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:33.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:33.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:33.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:33.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:33.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:33.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:33.484+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:33.485450+0000) 2024-02-20T09:15:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:33.758+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:33.759048+0000) 2024-02-20T09:15:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:33.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:33.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:33.715+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:33.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:33.716+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:34.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:34.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:34.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:34.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:34.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:34.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:34.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:34.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:34.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:34.243+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:34.244546+0000) 2024-02-20T09:15:34.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:34.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:34.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:34.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:34.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:34.135+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:34.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:34.135+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:34.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:34.488+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:34.488968+0000) 2024-02-20T09:15:34.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:34.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:34.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:34.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:34.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:34.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:34.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:34.760+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:34.761487+0000) 2024-02-20T09:15:35.249 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:35.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:35.249 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:35.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:35.249 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:35.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:35.250 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:35.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:35.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:35.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:35.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:35.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:35.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:35.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:35.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:35.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:35.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:35.490+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:35.491537+0000) 2024-02-20T09:15:35.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:35.247+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:35.248289+0000) 2024-02-20T09:15:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:35.763+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:35.763959+0000) 2024-02-20T09:15:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:35.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:35.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:35.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:35.725+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:36.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:36.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:36.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:36.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:36.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:36.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:36.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:36.140+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:36.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:36.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:36.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:36.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:36.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:36.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:36.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:36.148+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:36.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:36.248+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.249436+0000) 2024-02-20T09:15:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:36.491+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.492563+0000) 2024-02-20T09:15:37.020 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:36.896+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.897311+0000) 2024-02-20T09:15:37.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:36.763+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.764839+0000) 2024-02-20T09:15:37.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:36.897+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.898144+0000) 2024-02-20T09:15:37.021 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:36.899+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59804 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:37.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:36.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:37.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:36.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:37.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:36.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:37.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:36.735+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:37.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:36.897+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.897957+0000) 2024-02-20T09:15:37.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:37.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:37.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:37.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:37.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:37.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:37.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:37.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:37.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:37.248+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:37.249645+0000) 2024-02-20T09:15:37.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:37.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:37.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:37.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:37.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:37.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:37.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:37.168+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:37.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:37.491+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:37.492745+0000) 2024-02-20T09:15:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:37.764+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:37.765008+0000) 2024-02-20T09:15:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:37.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:37.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:37.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:37.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:38.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:38.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:38.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:38.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:38.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:38.198+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:38.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:38.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:38.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:38.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:38.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:38.250+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:38.251710+0000) 2024-02-20T09:15:38.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:38.495+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:38.496266+0000) 2024-02-20T09:15:38.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:38.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:38.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:38.766+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:38.767833+0000) 2024-02-20T09:15:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:38.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:38.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:38.709+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:39.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:39.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:39.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:39.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:39.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:39.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:39.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:39.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:39.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:39.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:39.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:39.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:39.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:39.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:39.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:39.173+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:39.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:39.253+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:39.254213+0000) 2024-02-20T09:15:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:39.498+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:39.499029+0000) 2024-02-20T09:15:40.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:39.769+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:39.770017+0000) 2024-02-20T09:15:40.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:39.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:40.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:39.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:40.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:39.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:40.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:39.737+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:40.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:40.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:40.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:40.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:40.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:40.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:40.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:40.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:40.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:40.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:40.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:40.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:40.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:40.149+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:40.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:40.150+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:40.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:40.256+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:40.257828+0000) 2024-02-20T09:15:40.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:40.500+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:40.501604+0000) 2024-02-20T09:15:40.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:40.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:40.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:40.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:40.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:40.710+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:40.711+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:40.772+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:40.773404+0000) 2024-02-20T09:15:41.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:41.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:41.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:41.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:41.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:41.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:41.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:41.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:41.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:41.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:41.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:41.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:41.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:41.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:41.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:41.195+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:41.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:41.258+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.259296+0000) 2024-02-20T09:15:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:41.500+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.502756+0000) 2024-02-20T09:15:42.028 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:41.906+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59842 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:42.028 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:41.772+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.774535+0000) 2024-02-20T09:15:42.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:41.903+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.905564+0000) 2024-02-20T09:15:42.029 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:41.902+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.904821+0000) 2024-02-20T09:15:42.029 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:41.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:42.029 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:41.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:42.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:41.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:42.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:41.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:42.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:41.902+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.904861+0000) 2024-02-20T09:15:42.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:42.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:42.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:42.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:42.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:42.209+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:42.280 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:42.210+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:42.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:42.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:42.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:42.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:42.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:42.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:42.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:42.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:42.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:42.257+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:42.259484+0000) 2024-02-20T09:15:42.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:42.501+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:42.502957+0000) 2024-02-20T09:15:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:42.772+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:42.774714+0000) 2024-02-20T09:15:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:42.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:42.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:42.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:42.741+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:43.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:43.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:43.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:43.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:43.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:43.259+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:43.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:43.260+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:43.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:43.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:43.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:43.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:43.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:43.158+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:43.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:43.158+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:43.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:43.260+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:43.262798+0000) 2024-02-20T09:15:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:43.504+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:43.506523+0000) 2024-02-20T09:15:44.114 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:43.776+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:43.778199+0000) 2024-02-20T09:15:44.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:43.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:44.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:43.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:44.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:43.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:44.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:43.735+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:44.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:44.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:44.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:44.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:44.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:44.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:44.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:44.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:44.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:44.264+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:44.266186+0000) 2024-02-20T09:15:44.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:44.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:44.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:44.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:44.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:44.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:44.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:44.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:44.506+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:44.508760+0000) 2024-02-20T09:15:45.103 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:44.779+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:44.780498+0000) 2024-02-20T09:15:45.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:44.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:45.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:44.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:45.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:44.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:45.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:44.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:45.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:45.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:45.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:45.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:45.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:45.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:45.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:45.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:45.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:45.268+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:45.269675+0000) 2024-02-20T09:15:45.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:45.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:45.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:45.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:45.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:45.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:45.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:45.299+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:45.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:45.510+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:45.511034+0000) 2024-02-20T09:15:46.080 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:45.782+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:45.783097+0000) 2024-02-20T09:15:46.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:45.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:46.080 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:45.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:46.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:45.726+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:46.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:45.727+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:46.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:46.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:46.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:46.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:46.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:46.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:46.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:46.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:46.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:46.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:46.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:46.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:46.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:46.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:46.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:46.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:46.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:46.269+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.270608+0000) 2024-02-20T09:15:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:46.511+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.512335+0000) 2024-02-20T09:15:47.033 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:46.783+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.783953+0000) 2024-02-20T09:15:47.034 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:46.912+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.913625+0000) 2024-02-20T09:15:47.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:46.914+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59883 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:47.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:46.912+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.913080+0000) 2024-02-20T09:15:47.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:46.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:47.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:46.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:47.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:46.740+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:47.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:46.742+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:47.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:46.911+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.912045+0000) 2024-02-20T09:15:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:47.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:47.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:47.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:47.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:47.269+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:47.270814+0000) 2024-02-20T09:15:47.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:47.511+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:47.512545+0000) 2024-02-20T09:15:47.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:47.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:47.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:47.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:47.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:47.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:47.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:47.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:47.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:47.783+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:47.784123+0000) 2024-02-20T09:15:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:47.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:47.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:47.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:47.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:48.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:48.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:48.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:48.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:48.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:48.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:48.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:48.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:48.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:48.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:48.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:48.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:48.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:48.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:48.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:48.076+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:48.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:48.272+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:48.273126+0000) 2024-02-20T09:15:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:48.514+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:48.515323+0000) 2024-02-20T09:15:49.062 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:48.785+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:48.786617+0000) 2024-02-20T09:15:49.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:48.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:49.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:48.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:49.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:48.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:49.063 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:48.773+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:49.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:49.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:49.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:49.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:49.063 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:49.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:49.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:49.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:49.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:49.275+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:49.276502+0000) 2024-02-20T09:15:49.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:49.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:49.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:49.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:49.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:49.294+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:49.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:49.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:49.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:49.517+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:49.518662+0000) 2024-02-20T09:15:50.018 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:49.789+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:49.789943+0000) 2024-02-20T09:15:50.019 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:49.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:50.019 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:49.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:50.019 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:49.758+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:50.019 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:49.759+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:50.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:50.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:50.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:50.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:50.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:50.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:50.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:50.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:50.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:50.521+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:50.522082+0000) 2024-02-20T09:15:50.604 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:50.277+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:50.278706+0000) 2024-02-20T09:15:50.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:50.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:50.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:50.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:50.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:50.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:50.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:50.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:50.792+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:50.793530+0000) 2024-02-20T09:15:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:50.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:50.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:50.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:50.714+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:51.280 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:51.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:51.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:51.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:51.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:51.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:51.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:51.013+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:51.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:51.522+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.523599+0000) 2024-02-20T09:15:51.604 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:51.278+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.279718+0000) 2024-02-20T09:15:51.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:51.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:51.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:51.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:51.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:51.290+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:51.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:51.291+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:51.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:51.793+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.794491+0000) 2024-02-20T09:15:51.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:51.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:51.918 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:51.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:51.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:51.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:51.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:51.705+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:52.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:51.922+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59921 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:51.919+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.920446+0000) 2024-02-20T09:15:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:51.918+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.919465+0000) 2024-02-20T09:15:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:51.919+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.920900+0000) 2024-02-20T09:15:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:52.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:52.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:52.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:52.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:52.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:52.278+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:52.279906+0000) 2024-02-20T09:15:52.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:52.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:52.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:52.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:52.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:52.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:52.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:52.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:52.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:52.522+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:52.523765+0000) 2024-02-20T09:15:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:52.659+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:52.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:52.660+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:52.661+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:53.102 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:52.793+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:52.794687+0000) 2024-02-20T09:15:53.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:53.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:53.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:53.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:53.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:53.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:53.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:53.296+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:53.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:53.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:53.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:53.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:53.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:53.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:53.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:53.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:53.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:53.281+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:53.282458+0000) 2024-02-20T09:15:53.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:53.526+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:53.527036+0000) 2024-02-20T09:15:53.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:53.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:53.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:53.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:53.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:53.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:53.796+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:53.797271+0000) 2024-02-20T09:15:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:53.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:54.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:54.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:54.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:54.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:54.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:54.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:54.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:54.311+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:54.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:54.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:54.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:54.114+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:54.284+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:54.285138+0000) 2024-02-20T09:15:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:54.528+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:54.529185+0000) 2024-02-20T09:15:55.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:54.798+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:54.799882+0000) 2024-02-20T09:15:55.107 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:54.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:55.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:54.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:55.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:54.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:55.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:54.723+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:55.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:55.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:55.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:55.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:55.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:55.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:55.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:55.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:55.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:55.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:55.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:55.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:55.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:55.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:55.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:55.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:55.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:55.286+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:55.287343+0000) 2024-02-20T09:15:55.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:55.531+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:55.531974+0000) 2024-02-20T09:15:55.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:55.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:55.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:55.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:55.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:55.707+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:55.801+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:55.802037+0000) 2024-02-20T09:15:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:55.707+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:56.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:56.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:56.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:56.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:56.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:56.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:56.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:56.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:56.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:56.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:56.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:56.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:56.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:56.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:56.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:56.094+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:56.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:56.287+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.288908+0000) 2024-02-20T09:15:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:56.531+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.532873+0000) 2024-02-20T09:15:57.051 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:15:56.930+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 59962 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:15:57.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:56.802+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.802950+0000) 2024-02-20T09:15:57.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:56.927+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.928335+0000) 2024-02-20T09:15:57.052 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:56.927+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.928372+0000) 2024-02-20T09:15:57.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:56.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:57.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:56.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:57.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:56.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:57.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:56.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:57.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:56.926+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.927796+0000) 2024-02-20T09:15:57.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:57.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:57.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:57.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:57.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:57.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:57.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:57.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:57.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:57.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:57.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:57.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:57.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:57.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:57.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:57.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:57.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:57.288+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:57.289096+0000) 2024-02-20T09:15:57.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:57.532+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:57.533035+0000) 2024-02-20T09:15:58.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:57.802+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:57.803129+0000) 2024-02-20T09:15:58.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:57.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:58.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:57.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:58.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:57.742+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:58.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:57.743+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:58.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:58.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:58.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:58.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:58.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:58.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:58.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:58.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:58.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:58.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:58.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:58.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:58.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:58.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:58.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:58.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:58.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:58.290+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:58.291541+0000) 2024-02-20T09:15:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:58.534+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:58.535577+0000) 2024-02-20T09:15:59.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:58.806+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:58.806975+0000) 2024-02-20T09:15:59.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:58.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:59.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:58.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:59.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:58.769+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:15:59.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:58.770+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:59.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:59.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:59.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:59.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:59.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:59.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:15:59.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:59.317+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:59.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:59.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:59.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:59.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:59.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:59.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:15:59.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:59.119+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:15:59.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:15:59.293+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:59.293954+0000) 2024-02-20T09:15:59.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:59.538+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:59.539084+0000) 2024-02-20T09:16:00.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:15:59.809+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:59.810556+0000) 2024-02-20T09:16:00.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:59.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:00.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:59.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:00.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:59.778+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:00.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:15:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:15:59.779+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:00.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:00.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:00.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:00.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:00.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:00.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:00.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:00.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:00.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:00.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:00.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:00.295+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:00.296311+0000) 2024-02-20T09:16:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:00.540+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:00.541729+0000) 2024-02-20T09:16:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:00.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:01.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:00.812+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:00.813188+0000) 2024-02-20T09:16:01.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:00.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:01.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:00.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:01.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:00.790+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:01.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:01.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:01.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:01.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:01.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:01.263+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:01.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:01.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:01.207+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:01.208+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:01.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:01.296+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.297294+0000) 2024-02-20T09:16:01.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:01.541+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.542737+0000) 2024-02-20T09:16:02.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:01.937+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60000 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:01.934+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.935843+0000) 2024-02-20T09:16:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:01.813+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.814523+0000) 2024-02-20T09:16:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:01.934+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.935622+0000) 2024-02-20T09:16:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:01.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:01.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:02.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:01.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:02.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:01.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:02.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:01.934+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.935331+0000) 2024-02-20T09:16:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:02.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:02.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:02.251+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:02.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:02.252+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:02.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:02.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:02.248+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:02.249+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:02.296+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:02.297480+0000) 2024-02-20T09:16:02.815 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:02.541+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:02.542926+0000) 2024-02-20T09:16:03.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:02.813+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:02.814726+0000) 2024-02-20T09:16:03.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:02.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:03.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:02.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:03.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:02.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:03.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:02.849+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:03.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:03.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:03.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:03.216+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:03.217+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:03.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:03.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:03.274+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:03.275+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:03.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:03.299+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:03.300266+0000) 2024-02-20T09:16:03.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:03.544+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:03.545288+0000) 2024-02-20T09:16:04.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:03.816+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:03.817111+0000) 2024-02-20T09:16:04.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:03.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:04.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:03.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:04.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:03.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:04.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:03.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:04.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:04.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:04.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:04.175+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:04.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:04.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:04.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:04.279+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:04.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:04.301+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:04.302643+0000) 2024-02-20T09:16:04.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:04.547+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:04.548042+0000) 2024-02-20T09:16:05.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:04.818+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:04.819789+0000) 2024-02-20T09:16:05.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:05.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:05.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:05.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:05.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:04.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:05.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:04.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:05.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:04.831+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:05.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:04.833+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:05.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:05.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:05.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:05.144+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:05.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:05.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:05.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:05.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:05.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:05.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:05.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:05.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:05.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:05.305+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:05.306113+0000) 2024-02-20T09:16:05.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:05.550+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:05.551678+0000) 2024-02-20T09:16:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:05.821+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:05.822640+0000) 2024-02-20T09:16:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:05.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:05.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:05.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:05.866+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:06.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:06.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:06.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:06.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:06.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:06.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:06.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:06.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:06.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:06.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:06.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:06.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:06.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:06.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:06.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:06.289+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:06.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:06.306+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.307681+0000) 2024-02-20T09:16:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:06.552+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.553286+0000) 2024-02-20T09:16:07.115 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:06.944+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60041 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:07.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:06.942+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.943500+0000) 2024-02-20T09:16:07.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:06.823+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.823974+0000) 2024-02-20T09:16:07.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:06.942+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.943438+0000) 2024-02-20T09:16:07.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:06.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:07.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:06.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:07.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:06.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:07.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:06.840+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:07.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:06.942+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.943581+0000) 2024-02-20T09:16:07.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:07.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:07.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:07.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:07.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:07.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:07.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:07.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:07.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:07.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:07.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:07.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:07.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:07.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:07.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:07.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:07.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:07.306+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:07.307922+0000) 2024-02-20T09:16:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:07.552+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:07.553509+0000) 2024-02-20T09:16:08.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:07.823+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:07.824151+0000) 2024-02-20T09:16:08.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:07.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:08.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:07.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:08.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:07.858+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:08.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:07.859+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:08.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:08.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:08.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:08.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:08.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:08.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:08.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:08.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:08.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:08.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:08.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:08.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:08.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:08.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:08.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:08.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:08.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:08.310+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:08.311203+0000) 2024-02-20T09:16:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:08.554+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:08.555538+0000) 2024-02-20T09:16:09.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:08.825+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:08.826544+0000) 2024-02-20T09:16:09.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:09.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:09.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:09.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:09.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:08.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:09.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:08.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:09.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:08.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:09.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:08.897+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:09.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:09.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:09.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:09.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:09.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:09.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:09.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:09.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:09.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:09.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:09.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:09.269+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:09.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:09.313+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:09.314777+0000) 2024-02-20T09:16:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:09.558+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:09.559118+0000) 2024-02-20T09:16:10.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:09.828+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:09.829039+0000) 2024-02-20T09:16:10.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:09.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:10.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:09.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:10.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:09.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:10.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:09.932+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:10.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:10.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:10.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:10.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:10.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:10.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:10.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:10.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:10.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:10.268+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:10.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:10.317+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:10.318124+0000) 2024-02-20T09:16:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:10.560+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:10.561683+0000) 2024-02-20T09:16:11.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:10.830+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:10.831758+0000) 2024-02-20T09:16:11.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:10.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:11.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:10.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:11.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:10.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:11.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:10.921+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:11.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:11.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:11.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:11.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:11.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:11.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:11.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:11.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:11.281+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:11.318+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.319029+0000) 2024-02-20T09:16:11.833 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:11.561+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.562730+0000) 2024-02-20T09:16:12.195 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:11.953+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60079 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:12.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:11.949+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.950069+0000) 2024-02-20T09:16:12.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:11.831+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.832804+0000) 2024-02-20T09:16:12.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:11.948+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.949667+0000) 2024-02-20T09:16:12.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:12.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:12.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:12.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:12.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:12.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:12.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:11.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:12.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:11.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:12.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:11.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:12.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:11.881+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:12.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:11.950+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.951015+0000) 2024-02-20T09:16:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:12.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:12.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:12.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:12.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:12.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:12.257+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:12.318+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:12.319223+0000) 2024-02-20T09:16:12.833 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:12.561+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:12.562892+0000) 2024-02-20T09:16:13.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:12.832+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:12.832979+0000) 2024-02-20T09:16:13.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:13.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:13.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:12.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:13.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:12.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:13.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:12.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:13.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:12.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:13.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:13.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:13.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:13.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:13.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:13.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:13.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:13.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:13.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:13.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:13.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:13.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:13.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:13.292+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:13.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:13.321+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:13.321968+0000) 2024-02-20T09:16:13.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:13.565+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:13.565968+0000) 2024-02-20T09:16:14.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:13.835+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:13.836509+0000) 2024-02-20T09:16:14.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:14.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:14.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:14.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:14.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:13.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:14.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:13.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:14.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:13.877+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:14.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:13.877+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:14.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:14.139+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:14.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:14.140+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:14.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:14.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:14.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:14.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:14.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:14.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:14.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:14.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:14.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:14.323+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:14.324056+0000) 2024-02-20T09:16:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:14.567+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:14.568416+0000) 2024-02-20T09:16:15.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:14.839+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:14.840080+0000) 2024-02-20T09:16:15.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:14.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:15.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:14.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:15.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:14.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:15.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:14.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:15.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:15.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:15.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:15.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:15.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:15.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:15.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:15.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:15.414 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:15.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:15.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:15.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:15.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:15.229+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:15.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:15.230+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:15.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:15.325+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:15.326478+0000) 2024-02-20T09:16:15.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:15.569+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:15.570839+0000) 2024-02-20T09:16:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:15.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:15.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:15.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:15.927+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:16.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:15.841+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:15.842532+0000) 2024-02-20T09:16:16.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:16.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:16.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:16.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:16.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:16.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:16.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:16.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:16.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:16.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:16.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:16.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:16.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:16.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:16.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.327690+0000) 2024-02-20T09:16:16.844 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:16.571+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.572221+0000) 2024-02-20T09:16:17.095 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:16.961+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60120 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:17.095 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:16.959+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.960100+0000) 2024-02-20T09:16:17.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:16.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:17.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:16.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:17.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:16.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:17.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:16.917+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:17.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:16.958+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.959731+0000) 2024-02-20T09:16:17.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:16.842+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.843776+0000) 2024-02-20T09:16:17.096 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:16.958+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.959735+0000) 2024-02-20T09:16:17.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:17.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:17.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:17.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:17.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:17.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:17.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:17.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:17.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:17.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:17.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:17.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:17.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:17.164+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:17.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:17.165+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:17.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:17.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:17.327890+0000) 2024-02-20T09:16:17.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:17.571+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:17.572408+0000) 2024-02-20T09:16:18.129 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:17.843+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:17.844092+0000) 2024-02-20T09:16:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:17.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:17.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:17.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:17.897+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:18.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:18.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:18.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:18.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:18.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:18.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:18.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:18.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:18.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:18.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:18.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:18.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:18.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:18.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:18.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:18.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:18.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:18.330+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:18.331313+0000) 2024-02-20T09:16:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:18.574+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:18.574971+0000) 2024-02-20T09:16:19.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:18.845+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:18.846149+0000) 2024-02-20T09:16:19.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:18.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:19.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:18.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:19.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:18.921+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:19.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:18.921+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:19.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:19.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:19.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:19.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:19.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:19.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:19.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:19.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:19.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:19.333+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:19.334609+0000) 2024-02-20T09:16:19.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:19.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:19.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:19.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:19.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:19.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:19.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:19.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:19.577+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:19.578610+0000) 2024-02-20T09:16:20.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:19.847+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:19.848614+0000) 2024-02-20T09:16:20.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:19.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:20.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:19.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:20.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:19.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:20.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:19.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:20.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:20.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:20.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:20.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:20.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:20.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:20.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:20.234+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:20.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:20.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:20.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:20.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:20.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:20.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:20.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:20.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:20.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:20.337+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:20.338116+0000) 2024-02-20T09:16:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:20.581+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:20.582188+0000) 2024-02-20T09:16:21.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:20.850+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:20.851216+0000) 2024-02-20T09:16:21.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:20.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:21.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:20.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:21.112 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:20.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:21.112 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:20.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:21.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:21.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:21.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:21.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:21.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:21.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:21.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:21.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:21.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:21.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:21.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:21.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:21.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:21.109+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:21.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:21.110+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:21.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:21.338+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.339711+0000) 2024-02-20T09:16:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:21.582+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.583813+0000) 2024-02-20T09:16:22.107 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:21.968+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60158 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:22.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:21.851+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.852738+0000) 2024-02-20T09:16:22.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:21.966+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.967888+0000) 2024-02-20T09:16:22.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:21.965+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.966728+0000) 2024-02-20T09:16:22.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:22.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:22.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:22.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:22.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:21.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:22.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:21.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:22.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:21.911+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:22.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:21.911+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:22.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:21.965+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.966658+0000) 2024-02-20T09:16:22.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:22.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:22.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:22.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:22.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:22.338+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:22.339913+0000) 2024-02-20T09:16:22.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:22.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:22.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:22.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:22.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:22.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:22.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:22.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:22.583+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:22.584028+0000) 2024-02-20T09:16:23.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:22.852+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:22.852931+0000) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:23.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:23.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:23.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:23.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:22.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:22.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:22.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:23.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:22.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:23.495 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:23.341+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:23.342372+0000) 2024-02-20T09:16:23.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:23.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:23.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:23.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:23.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:23.240+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:23.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:23.241+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:23.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:23.585+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:23.586164+0000) 2024-02-20T09:16:24.075 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:23.854+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:23.855504+0000) 2024-02-20T09:16:24.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:23.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:24.075 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:23.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:24.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:23.893+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:24.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:23.894+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:24.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:24.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:24.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:24.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:24.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:24.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:24.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:24.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:24.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:24.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:24.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:24.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:24.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:24.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:24.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:24.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:24.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:24.588+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:24.589309+0000) 2024-02-20T09:16:24.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:24.343+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:24.344795+0000) 2024-02-20T09:16:25.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:24.857+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:24.858052+0000) 2024-02-20T09:16:25.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:25.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:25.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:25.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:25.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:25.205+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:25.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:24.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:25.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:24.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:25.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:24.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:25.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:24.937+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:25.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:25.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:25.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:25.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:25.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:25.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:25.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:25.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:25.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:25.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:25.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:25.346+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:25.347213+0000) 2024-02-20T09:16:25.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:25.590+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:25.591851+0000) 2024-02-20T09:16:26.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:25.859+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:25.860548+0000) 2024-02-20T09:16:26.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:26.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:26.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:25.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:25.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:25.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:25.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:26.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:26.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:26.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:26.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:26.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:26.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:26.347+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.348629+0000) 2024-02-20T09:16:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:26.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:26.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:26.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:26.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:26.591+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.592862+0000) 2024-02-20T09:16:27.130 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:26.975+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60199 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:27.131 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:26.861+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.862217+0000) 2024-02-20T09:16:27.131 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:26.972+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.973635+0000) 2024-02-20T09:16:27.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:26.973+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.974560+0000) 2024-02-20T09:16:27.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:26.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:27.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:26.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:27.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:26.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:27.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:26.883+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:27.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:26.973+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.974093+0000) 2024-02-20T09:16:27.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:27.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:27.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:27.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:27.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:27.223+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:27.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:27.223+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:27.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:27.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:27.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:27.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:27.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:27.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:27.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:27.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:27.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:27.347+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:27.348807+0000) 2024-02-20T09:16:27.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:27.592+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:27.593027+0000) 2024-02-20T09:16:28.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:27.861+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:27.862417+0000) 2024-02-20T09:16:28.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:27.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:28.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:27.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:28.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:27.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:28.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:27.842+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:28.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:28.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:28.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:28.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:28.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:28.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:28.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:28.258+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:28.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:28.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:28.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:28.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:28.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:28.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:28.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:28.143+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:28.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:28.349+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:28.350808+0000) 2024-02-20T09:16:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:28.594+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:28.595543+0000) 2024-02-20T09:16:29.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:28.863+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:28.864831+0000) 2024-02-20T09:16:29.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:28.833+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:29.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:28.833+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:29.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:28.833+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:29.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:28.834+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:29.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:29.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:29.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:29.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:29.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:29.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:29.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:29.158+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:29.415 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:29.353+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:29.354109+0000) 2024-02-20T09:16:29.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:29.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:29.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:29.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:29.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:29.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:29.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:29.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:29.597+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:29.598904+0000) 2024-02-20T09:16:30.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:29.866+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:29.867070+0000) 2024-02-20T09:16:30.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:29.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:30.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:29.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:30.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:29.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:30.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:29.831+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:30.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:30.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:30.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:30.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:30.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:30.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:30.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:30.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:30.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:30.356+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:30.357474+0000) 2024-02-20T09:16:30.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:30.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:30.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:30.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:30.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:30.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:30.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:30.263+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:30.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:30.599+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:30.600666+0000) 2024-02-20T09:16:31.094 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:30.868+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:30.869572+0000) 2024-02-20T09:16:31.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:30.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:31.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:30.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:31.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:30.841+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:31.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:30.842+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:31.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:31.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:31.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:31.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:31.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:31.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:31.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:31.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:31.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:31.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:31.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:31.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:31.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:31.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:31.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:31.093+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:31.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:31.357+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.358638+0000) 2024-02-20T09:16:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:31.601+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.602256+0000) 2024-02-20T09:16:32.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:31.983+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60237 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:31.870+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.871235+0000) 2024-02-20T09:16:32.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:31.981+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.982752+0000) 2024-02-20T09:16:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:31.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:31.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:31.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:31.879+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:32.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:31.981+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.982052+0000) 2024-02-20T09:16:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:31.980+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.981000+0000) 2024-02-20T09:16:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:32.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:32.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:32.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:32.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:32.603 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:32.357+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:32.358833+0000) 2024-02-20T09:16:32.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:32.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:32.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:32.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:32.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:32.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:32.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:32.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:32.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:32.601+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:32.602438+0000) 2024-02-20T09:16:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:32.870+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:32.871442+0000) 2024-02-20T09:16:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:32.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:32.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:32.908+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:32.909+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:33.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:33.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:33.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:33.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:33.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:33.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:33.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:33.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:33.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:33.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:33.311+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:33.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:33.312+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:33.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:33.361+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:33.362207+0000) 2024-02-20T09:16:33.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:33.604+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:33.605053+0000) 2024-02-20T09:16:33.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:33.863+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:33.865 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:33.863+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:33.866 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:33.863+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:33.872+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:33.873622+0000) 2024-02-20T09:16:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:33.864+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:34.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:34.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:34.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:34.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:34.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:34.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:34.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:34.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:34.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:34.299+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:34.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:34.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:34.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:34.362+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:34.364714+0000) 2024-02-20T09:16:34.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:34.605+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:34.607907+0000) 2024-02-20T09:16:34.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:34.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:34.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:34.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:34.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:34.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:34.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:34.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:34.873+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:34.875854+0000) 2024-02-20T09:16:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:35.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:35.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:35.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:35.036+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:35.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:35.366+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:35.368407+0000) 2024-02-20T09:16:35.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:35.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:35.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:35.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:35.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:35.264+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:35.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:35.265+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:35.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:35.608+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:35.610553+0000) 2024-02-20T09:16:35.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:35.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:35.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:35.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:35.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:35.803+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:35.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:35.804+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:35.876+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:35.878634+0000) 2024-02-20T09:16:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:36.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:36.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:36.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:36.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:36.006+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:36.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:36.367+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.369652+0000) 2024-02-20T09:16:36.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:36.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:36.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:36.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:36.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:36.231+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:36.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:36.232+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:36.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:36.610+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.611981+0000) 2024-02-20T09:16:36.814 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:36.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:36.814 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:36.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:36.814 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:36.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:37.120 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:36.989+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60278 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:37.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:36.877+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.879815+0000) 2024-02-20T09:16:37.120 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:36.986+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.988516+0000) 2024-02-20T09:16:37.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:36.811+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:37.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:36.988+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.990123+0000) 2024-02-20T09:16:37.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:36.986+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.988866+0000) 2024-02-20T09:16:37.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:37.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:37.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:37.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:37.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:37.054+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:37.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:37.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:37.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:37.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:37.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:37.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:37.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:37.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:37.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:37.187+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:37.611+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:37.612149+0000) 2024-02-20T09:16:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:37.368+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:37.369874+0000) 2024-02-20T09:16:38.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:37.879+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:37.880068+0000) 2024-02-20T09:16:38.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:37.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:38.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:37.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:38.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:37.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:38.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:37.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:38.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:38.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:38.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:38.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.031+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:38.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.371+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:38.372500+0000) 2024-02-20T09:16:38.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:38.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:38.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:38.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:38.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:38.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:38.614+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:38.615773+0000) 2024-02-20T09:16:38.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:38.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:38.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:38.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:38.810 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:38.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:39.163 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:38.882+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:38.883177+0000) 2024-02-20T09:16:39.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:38.808+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:39.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:39.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:39.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:39.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:38.994+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:39.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:39.374+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:39.375117+0000) 2024-02-20T09:16:39.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:39.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:39.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:39.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:39.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:39.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:39.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:39.162+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:39.617+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:39.618249+0000) 2024-02-20T09:16:40.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:39.884+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:39.885241+0000) 2024-02-20T09:16:40.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:39.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:40.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:39.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:40.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:39.825+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:40.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:39.826+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:40.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:39.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:40.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:39.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:40.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:39.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:40.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:39.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:40.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:40.376+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:40.377637+0000) 2024-02-20T09:16:40.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:40.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:40.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:40.191+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:40.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:40.192+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:40.829 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:40.619+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:40.620853+0000) 2024-02-20T09:16:40.829 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:40.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:40.829 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:40.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:40.829 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:40.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:41.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:40.886+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:40.887836+0000) 2024-02-20T09:16:41.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:40.827+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:41.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:40.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:41.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:40.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:41.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:40.988+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:41.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:40.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:41.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:41.377+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.378831+0000) 2024-02-20T09:16:41.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:41.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:41.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:41.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:41.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:41.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:41.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:41.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:41.620+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.621876+0000) 2024-02-20T09:16:42.174 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:41.998+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60316 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:42.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:41.888+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.889261+0000) 2024-02-20T09:16:42.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:41.994+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.995885+0000) 2024-02-20T09:16:42.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:41.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:42.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:41.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:42.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:41.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:41.964+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:41.995+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.996338+0000) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:41.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:41.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:41.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:41.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:42.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:41.995+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.995990+0000) 2024-02-20T09:16:42.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:42.378+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:42.379032+0000) 2024-02-20T09:16:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:42.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:42.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:42.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:42.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:42.173+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:42.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:42.621+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:42.622066+0000) 2024-02-20T09:16:42.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:42.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:42.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:42.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:42.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:42.870+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:43.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:42.871+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:43.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:42.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:43.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:42.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:43.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:42.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:43.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:42.947+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:43.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:42.888+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:42.889438+0000) 2024-02-20T09:16:43.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:43.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:43.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:43.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:43.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:43.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:43.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:43.149+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:43.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:43.380+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:43.381807+0000) 2024-02-20T09:16:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:43.623+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:43.624527+0000) 2024-02-20T09:16:44.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:43.890+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:43.891415+0000) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:43.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:43.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:43.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:43.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:43.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:43.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:44.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:43.967+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:44.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:43.968+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:44.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:44.383+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:44.384141+0000) 2024-02-20T09:16:44.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:44.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:44.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:44.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:44.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:44.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:44.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:44.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:44.625+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:44.626574+0000) 2024-02-20T09:16:45.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:44.894+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:44.894954+0000) 2024-02-20T09:16:45.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:45.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:45.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:45.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:45.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:45.140+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:45.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:44.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:45.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:44.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:45.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:44.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:45.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:44.887+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:45.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:44.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:45.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:44.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:45.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:44.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:45.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:44.926+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:45.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:45.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:45.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:45.385+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:45.386205+0000) 2024-02-20T09:16:45.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:45.627+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:45.628762+0000) 2024-02-20T09:16:46.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:45.897+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:45.898305+0000) 2024-02-20T09:16:46.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:45.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:46.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:45.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:46.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:45.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:46.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:45.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:46.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:45.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:46.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:45.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:46.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:45.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:46.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:45.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:46.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:46.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.387402+0000) 2024-02-20T09:16:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:46.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:46.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:46.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:46.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:46.169+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:46.900 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:46.628+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.629783+0000) 2024-02-20T09:16:47.180 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:47.005+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60357 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:47.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:46.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:47.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:46.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:47.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:46.914+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:47.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:46.914+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:47.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:47.003+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.004435+0000) 2024-02-20T09:16:47.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:46.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:47.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:46.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:47.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:46.908+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:47.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:46.909+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:47.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:47.003+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.003974+0000) 2024-02-20T09:16:47.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:46.898+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.899604+0000) 2024-02-20T09:16:47.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:47.002+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.003028+0000) 2024-02-20T09:16:47.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:47.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:47.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:47.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:47.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:47.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:47.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:47.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:47.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:47.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.387609+0000) 2024-02-20T09:16:47.900 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:47.629+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.629961+0000) 2024-02-20T09:16:48.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:47.898+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.899802+0000) 2024-02-20T09:16:48.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:47.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:48.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:47.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:48.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:47.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:48.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:47.899+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:48.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:47.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:48.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:47.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:48.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:47.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:48.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:47.907+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:48.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:48.389+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:48.389958+0000) 2024-02-20T09:16:48.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:48.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:48.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:48.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:48.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:48.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:48.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:48.632+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:48.633289+0000) 2024-02-20T09:16:49.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:48.902+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:48.903085+0000) 2024-02-20T09:16:49.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:48.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:49.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:48.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:49.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:48.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:49.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:48.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:49.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:48.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:49.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:48.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:49.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:48.936+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:49.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:48.937+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:49.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:49.391+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:49.392114+0000) 2024-02-20T09:16:49.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:49.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:49.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:49.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:49.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:49.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:49.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:49.635+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:49.636727+0000) 2024-02-20T09:16:49.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:49.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:49.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:49.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:50.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:49.865+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:50.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:49.866+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:50.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:49.905+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:49.906376+0000) 2024-02-20T09:16:50.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:50.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:50.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:50.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:50.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:50.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:50.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:49.913+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:50.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:49.913+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:50.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:49.913+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:50.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:49.914+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:50.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:50.125+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:50.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:50.394+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:50.395461+0000) 2024-02-20T09:16:50.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:50.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:50.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:50.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:50.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:50.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:50.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:50.875+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:50.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:50.638+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:50.639330+0000) 2024-02-20T09:16:50.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:50.828+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:50.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:50.828+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:50.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:50.828+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:50.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:50.829+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:51.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:50.908+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:50.908971+0000) 2024-02-20T09:16:51.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:51.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:51.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:51.396+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.397086+0000) 2024-02-20T09:16:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:51.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:51.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:51.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:51.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:51.640+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.640955+0000) 2024-02-20T09:16:52.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:52.013+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60395 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:52.033 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:51.909+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.910616+0000) 2024-02-20T09:16:52.033 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:52.010+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.011394+0000) 2024-02-20T09:16:52.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:51.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:52.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:51.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:52.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:51.872+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:51.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:52.010+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.011062+0000) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:51.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:51.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:51.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:51.839+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:52.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:52.010+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.011628+0000) 2024-02-20T09:16:52.287 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:52.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:52.287 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:52.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:52.287 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:52.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:52.287 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:52.128+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:52.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:52.396+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.397281+0000) 2024-02-20T09:16:52.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:52.640+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.641138+0000) 2024-02-20T09:16:52.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:52.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:52.902 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:52.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:52.902 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:52.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:52.902 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:52.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:52.909+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.910795+0000) 2024-02-20T09:16:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:53.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:53.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:53.116+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:53.117+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:52.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:52.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:52.899+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:52.900+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:53.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:53.399+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:53.400640+0000) 2024-02-20T09:16:53.897 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:53.642+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:53.643298+0000) 2024-02-20T09:16:53.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:53.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:53.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:53.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:53.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:53.856+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:53.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:53.857+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:54.158 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:53.913+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:53.914027+0000) 2024-02-20T09:16:54.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:53.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:54.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:53.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:54.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:53.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:54.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:53.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:54.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:54.403+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:54.403958+0000) 2024-02-20T09:16:54.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:54.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:54.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:54.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:54.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:54.156+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:54.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:54.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:54.644+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:54.645450+0000) 2024-02-20T09:16:55.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:54.916+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:54.917612+0000) 2024-02-20T09:16:55.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:55.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:55.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:55.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:55.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:54.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:55.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:54.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:55.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:54.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:55.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:54.913+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:55.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:54.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:55.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:54.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:55.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:54.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:55.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:54.883+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:55.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:55.405+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:55.406103+0000) 2024-02-20T09:16:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:55.188+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:55.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:55.189+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:55.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:55.646+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:55.647668+0000) 2024-02-20T09:16:56.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:55.920+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:55.921197+0000) 2024-02-20T09:16:56.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:56.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:56.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:56.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:55.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:55.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:55.906+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:55.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:55.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:55.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:56.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:55.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:56.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:55.903+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:56.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:56.406+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.407227+0000) 2024-02-20T09:16:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:56.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:56.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:56.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:56.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:56.648+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.649362+0000) 2024-02-20T09:16:57.180 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:16:57.020+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60436 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:16:57.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:56.921+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.922289+0000) 2024-02-20T09:16:57.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:57.018+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.019789+0000) 2024-02-20T09:16:57.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:56.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:57.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:56.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:57.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:56.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:57.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:56.905+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:57.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:57.018+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.019490+0000) 2024-02-20T09:16:57.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:56.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:57.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:56.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:57.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:56.912+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:57.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:56.913+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:57.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:57.018+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.019021+0000) 2024-02-20T09:16:57.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:57.406+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.407428+0000) 2024-02-20T09:16:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:57.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:57.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:57.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:57.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:57.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:57.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:57.648+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.649563+0000) 2024-02-20T09:16:57.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:57.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:57.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:57.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:57.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:57.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:58.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:57.902+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:58.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:57.921+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.922464+0000) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:58.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:58.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:58.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:58.152+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:57.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:57.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:58.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:57.948+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:58.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:57.949+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:58.650+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:58.651666+0000) 2024-02-20T09:16:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:58.410+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:58.410968+0000) 2024-02-20T09:16:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:58.923+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:58.924851+0000) 2024-02-20T09:16:59.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:59.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:59.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:59.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:16:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:59.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:58.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:58.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:58.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:16:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:58.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:16:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:58.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:58.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:58.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:16:59.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:58.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:59.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:59.412+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:59.413566+0000) 2024-02-20T09:16:59.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:59.653+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:59.654167+0000) 2024-02-20T09:17:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:16:59.926+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:59.927588+0000) 2024-02-20T09:17:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:00.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:00.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:00.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:00.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:00.149+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:59.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:59.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:59.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:16:59.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:59.928+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:59.928+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:59.928+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:16:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:16:59.929+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:00.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:00.414+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:00.415766+0000) 2024-02-20T09:17:00.893 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:00.656+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:00.657757+0000) 2024-02-20T09:17:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:00.929+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:00.930158+0000) 2024-02-20T09:17:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:01.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:01.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:01.123+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:01.124+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:00.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:00.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:00.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:01.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:00.911+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:00.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:00.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:00.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:00.892+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:01.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:01.415+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.416879+0000) 2024-02-20T09:17:01.894 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:01.657+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.658889+0000) 2024-02-20T09:17:02.149 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:02.027+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60474 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:02.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:01.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:02.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:01.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:02.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:01.937+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:02.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:01.937+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:02.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:02.025+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.026808+0000) 2024-02-20T09:17:02.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:01.930+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.931283+0000) 2024-02-20T09:17:02.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:02.024+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.025866+0000) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:02.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:02.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:02.086+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:02.087+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:01.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:01.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:02.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:01.893+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:02.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:01.894+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:02.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:02.024+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.025153+0000) 2024-02-20T09:17:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:02.416+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.417068+0000) 2024-02-20T09:17:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:02.658+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.659029+0000) 2024-02-20T09:17:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:02.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:02.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:02.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:02.946+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:02.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:02.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:02.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:02.889+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:02.930+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.931484+0000) 2024-02-20T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:03.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:03.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:03.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:03.071+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:03.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:03.418+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:03.419648+0000) 2024-02-20T09:17:03.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:03.660+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:03.661246+0000) 2024-02-20T09:17:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:03.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:03.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:03.919+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:03.920+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:03.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:03.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:03.917+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:04.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:03.918+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:03.933+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:03.934826+0000) 2024-02-20T09:17:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:04.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:04.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:04.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:04.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:04.106+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:04.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:04.421+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:04.422171+0000) 2024-02-20T09:17:04.882 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:04.663+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:04.664862+0000) 2024-02-20T09:17:05.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:04.936+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:04.937025+0000) 2024-02-20T09:17:05.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:05.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:05.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:05.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:05.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:04.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:05.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:04.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:05.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:04.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:05.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:04.935+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:05.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:04.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:05.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:04.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:05.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:04.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:05.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:04.881+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:05.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:05.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:05.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:05.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:05.665+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:05.666860+0000) 2024-02-20T09:17:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:05.423+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:05.424615+0000) 2024-02-20T09:17:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:05.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:05.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:05.918+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:05.919+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:05.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:05.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:05.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:05.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:05.938+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:05.939101+0000) 2024-02-20T09:17:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:06.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:06.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:06.166+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:06.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:06.167+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:06.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:06.425+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.426432+0000) 2024-02-20T09:17:06.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:06.667+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.668584+0000) 2024-02-20T09:17:07.155 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:07.035+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60515 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:07.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:06.939+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.940518+0000) 2024-02-20T09:17:07.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:07.032+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.033638+0000) 2024-02-20T09:17:07.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:06.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:06.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:06.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:06.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:07.032+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.033489+0000) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:06.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:06.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:07.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:06.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:07.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:06.955+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:07.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:07.033+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.034070+0000) 2024-02-20T09:17:07.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:07.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:07.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:07.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:07.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:07.196+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:07.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:07.197+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:07.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:07.425+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.426620+0000) 2024-02-20T09:17:07.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:07.667+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.668788+0000) 2024-02-20T09:17:07.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:07.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:07.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:07.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:07.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:07.938+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:08.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:07.939+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.940701+0000) 2024-02-20T09:17:08.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:07.939+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:08.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:07.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:08.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:07.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:08.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:07.945+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:08.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:07.946+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:08.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:08.427+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:08.428849+0000) 2024-02-20T09:17:08.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:08.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:08.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:08.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:08.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:08.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:08.896 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:08.670+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:08.671421+0000) 2024-02-20T09:17:09.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:08.943+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:08.944190+0000) 2024-02-20T09:17:09.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:09.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:09.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:08.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:09.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:08.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:09.161 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:08.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:09.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:08.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:09.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:08.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:09.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:08.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:09.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:08.894+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:09.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:08.895+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:09.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:09.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:09.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:09.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:09.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:09.159+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:09.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:09.430+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:09.431096+0000) 2024-02-20T09:17:09.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:09.673+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:09.674926+0000) 2024-02-20T09:17:09.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:09.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:09.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:09.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:09.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:09.938+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:09.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:09.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:09.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:09.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:09.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:09.902+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:09.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:09.903+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:09.940+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:09.945+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:09.946645+0000) 2024-02-20T09:17:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:10.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:10.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:10.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:10.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:10.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:10.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:10.432+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:10.433716+0000) 2024-02-20T09:17:10.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:10.676+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:10.677182+0000) 2024-02-20T09:17:11.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:10.949+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:10.950126+0000) 2024-02-20T09:17:11.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:10.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:11.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:10.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:11.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:10.960+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:11.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:10.961+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:11.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:10.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:11.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:10.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:11.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:10.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:11.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:10.871+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:11.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:11.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:11.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:11.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:11.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:11.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:11.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:11.145+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:11.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:11.434+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.434955+0000) 2024-02-20T09:17:11.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:11.678+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.678969+0000) 2024-02-20T09:17:11.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:11.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:11.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:11.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:11.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:11.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:11.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:11.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:11.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:11.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:11.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:11.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:11.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:11.904+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:11.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:11.905+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:12.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:12.042+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60549 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:12.040+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.041047+0000) 2024-02-20T09:17:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:12.040+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.041088+0000) 2024-02-20T09:17:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:11.950+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.951407+0000) 2024-02-20T09:17:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:12.040+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.041264+0000) 2024-02-20T09:17:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:12.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:12.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:12.612 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:12.434+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.435141+0000) 2024-02-20T09:17:12.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:12.678+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.679182+0000) 2024-02-20T09:17:13.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:12.950+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.951596+0000) 2024-02-20T09:17:13.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:12.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:13.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:12.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:13.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:12.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:13.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:12.885+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:13.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:12.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:13.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:12.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:13.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:12.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:13.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:12.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:13.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:13.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:13.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:13.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:13.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:13.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:13.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:13.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:13.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:13.436+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:13.437547+0000) 2024-02-20T09:17:13.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:13.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:13.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:13.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:13.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:13.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:13.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:13.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:13.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:13.681+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:13.682091+0000) 2024-02-20T09:17:13.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:13.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:13.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:13.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:13.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:13.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:13.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:13.882+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:13.953+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:13.954107+0000) 2024-02-20T09:17:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:14.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:14.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:14.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:14.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:14.438+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:14.439799+0000) 2024-02-20T09:17:14.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:14.683+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:14.684561+0000) 2024-02-20T09:17:15.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:14.955+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:14.956719+0000) 2024-02-20T09:17:15.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:15.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:15.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:14.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:15.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:14.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:15.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:14.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:15.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:14.932+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:15.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:14.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:15.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:14.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:15.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:14.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:15.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:14.866+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:15.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:15.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:15.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:15.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:15.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:15.176+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:15.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:15.686+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:15.687029+0000) 2024-02-20T09:17:15.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:15.441+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:15.442326+0000) 2024-02-20T09:17:16.132 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:15.958+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:15.959179+0000) 2024-02-20T09:17:16.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:15.973+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:16.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:15.973+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:16.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:15.973+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:16.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:15.974+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:16.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:15.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:16.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:15.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:16.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:15.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:16.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:15.844+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:16.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:16.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:16.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:16.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:16.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:16.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:16.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:16.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:16.689 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:16.443+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.444130+0000) 2024-02-20T09:17:16.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:16.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:16.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:16.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:16.836+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:16.687+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.688654+0000) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:16.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:16.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:16.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:16.936+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:16.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:16.937+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:17.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:17.049+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60590 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:17.047+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.048798+0000) 2024-02-20T09:17:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:17.047+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.048784+0000) 2024-02-20T09:17:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:16.959+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.960603+0000) 2024-02-20T09:17:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:17.047+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.048829+0000) 2024-02-20T09:17:17.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:17.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:17.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:17.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:17.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:17.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:17.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:17.443+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.444329+0000) 2024-02-20T09:17:17.863 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:17.688+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.688945+0000) 2024-02-20T09:17:18.130 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:17.959+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.960831+0000) 2024-02-20T09:17:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:17.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:17.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:17.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:18.130 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:17.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:18.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:17.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:18.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:17.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:18.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:17.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:18.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:17.862+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:18.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:18.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:18.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:18.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:18.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:18.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:18.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:18.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:18.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:18.445+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:18.446749+0000) 2024-02-20T09:17:18.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:18.691+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:18.692472+0000) 2024-02-20T09:17:18.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:18.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:18.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:18.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:18.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:18.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:18.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:18.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:18.962+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:18.963394+0000) 2024-02-20T09:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:19.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:19.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:19.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:19.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:18.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:18.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:18.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:19.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:18.957+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:19.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:19.448+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:19.449440+0000) 2024-02-20T09:17:19.834 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:19.694+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:19.694952+0000) 2024-02-20T09:17:20.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:19.965+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:19.966118+0000) 2024-02-20T09:17:20.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:19.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:20.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:19.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:20.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:19.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:20.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:19.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:20.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:19.833+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:20.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:19.833+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:20.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:19.833+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:20.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:19.834+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:20.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:20.170+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:20.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:20.170+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:20.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:20.170+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:20.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:20.171+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:20.695+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:20.696925+0000) 2024-02-20T09:17:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:20.451+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:20.452019+0000) 2024-02-20T09:17:21.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:20.967+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:20.968697+0000) 2024-02-20T09:17:21.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:20.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:21.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:20.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:21.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:20.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:21.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:20.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:21.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:20.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:21.179 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:20.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:21.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:20.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:21.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:20.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:21.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:21.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:21.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:21.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:21.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:21.176+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:21.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:21.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:21.452+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.453250+0000) 2024-02-20T09:17:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:21.697+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.698749+0000) 2024-02-20T09:17:22.171 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:22.057+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60628 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:22.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:21.969+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.970541+0000) 2024-02-20T09:17:22.171 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:22.054+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.055128+0000) 2024-02-20T09:17:22.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:21.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:22.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:21.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:22.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:21.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:22.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:21.893+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:22.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:22.053+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.054858+0000) 2024-02-20T09:17:22.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:21.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:22.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:21.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:22.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:21.907+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:22.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:21.908+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:22.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:22.054+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.055370+0000) 2024-02-20T09:17:22.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:22.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:22.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:22.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:22.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:22.169+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:22.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:22.170+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:22.698+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.698957+0000) 2024-02-20T09:17:22.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:22.452+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.453464+0000) 2024-02-20T09:17:23.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:22.969+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.970719+0000) 2024-02-20T09:17:23.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:22.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:23.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:22.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:23.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:22.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:23.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:22.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:23.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:22.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:23.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:22.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:23.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:22.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:23.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:22.885+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:23.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:23.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:23.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:23.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:23.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:23.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:23.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:23.178+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:23.701+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:23.702426+0000) 2024-02-20T09:17:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:23.456+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:23.457133+0000) 2024-02-20T09:17:24.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:23.973+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:23.974028+0000) 2024-02-20T09:17:24.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:23.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:24.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:23.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:24.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:23.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:24.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:23.891+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:24.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:23.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:24.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:23.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:24.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:23.843+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:24.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:23.844+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:24.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:24.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:24.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:24.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:24.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:24.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:24.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:24.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:24.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:24.458+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:24.459785+0000) 2024-02-20T09:17:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:24.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:24.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:24.811+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:24.812+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:24.703+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:24.704927+0000) 2024-02-20T09:17:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:24.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:24.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:24.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:24.880+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:24.975+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:24.976121+0000) 2024-02-20T09:17:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:25.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:25.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:25.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:25.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:25.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:25.112+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:25.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:25.462+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:25.463053+0000) 2024-02-20T09:17:25.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:25.706+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:25.707249+0000) 2024-02-20T09:17:26.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:25.977+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:25.978410+0000) 2024-02-20T09:17:26.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:26.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:26.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:26.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:26.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:26.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:26.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:25.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:26.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:25.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:26.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:25.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:26.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:25.917+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:25.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:25.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:25.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:26.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:25.802+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:26.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:26.138+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:26.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:26.463+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.464805+0000) 2024-02-20T09:17:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:26.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:26.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:26.780+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:26.781+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:26.707+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.708430+0000) 2024-02-20T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:26.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:26.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:26.883+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:26.884+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:27.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:27.065+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60669 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:27.062+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.063913+0000) 2024-02-20T09:17:27.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:27.062+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.063038+0000) 2024-02-20T09:17:27.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:26.978+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.979562+0000) 2024-02-20T09:17:27.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:27.062+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.063813+0000) 2024-02-20T09:17:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:27.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:27.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:27.106+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:27.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:27.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:27.463+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.464973+0000) 2024-02-20T09:17:27.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:27.706+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.708780+0000) 2024-02-20T09:17:27.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:27.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:27.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:27.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:27.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:27.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:27.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:27.775+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:27.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:27.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:28.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:27.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:27.882+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:27.977+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.979766+0000) 2024-02-20T09:17:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:28.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:28.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:28.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:28.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:28.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:28.466+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:28.468374+0000) 2024-02-20T09:17:28.897 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:28.709+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:28.711807+0000) 2024-02-20T09:17:28.897 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:28.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:28.898 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:28.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:28.898 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:28.790+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:28.898 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:28.791+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:28.980+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:28.981954+0000) 2024-02-20T09:17:29.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:29.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:29.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:29.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:29.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:28.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:28.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:29.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:28.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:29.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:28.895+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:29.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:29.469+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:29.471071+0000) 2024-02-20T09:17:29.830 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:29.712+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:29.713950+0000) 2024-02-20T09:17:30.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:29.982+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:29.984436+0000) 2024-02-20T09:17:30.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:29.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:30.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:29.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:30.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:29.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:30.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:29.895+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:30.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:29.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:30.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:29.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:30.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:29.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:30.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:29.828+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:30.190+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:30.190+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:30.190+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:30.191+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:30.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:30.716+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:30.717502+0000) 2024-02-20T09:17:30.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:30.471+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:30.473651+0000) 2024-02-20T09:17:31.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:30.987+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:30.987977+0000) 2024-02-20T09:17:31.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:30.853+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:31.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:30.853+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:31.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:30.853+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:31.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:30.854+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:31.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:30.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:31.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:30.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:31.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:30.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:31.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:30.818+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:31.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:31.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:31.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:31.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:31.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:31.718+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.718970+0000) 2024-02-20T09:17:31.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:31.474+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.475571+0000) 2024-02-20T09:17:31.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:31.823+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:31.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:31.823+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:31.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:31.823+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:32.082 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:32.073+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60707 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:32.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:31.988+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.989556+0000) 2024-02-20T09:17:32.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:32.069+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.070496+0000) 2024-02-20T09:17:32.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:31.824+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:32.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:32.068+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.069806+0000) 2024-02-20T09:17:32.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:31.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:32.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:31.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:32.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:31.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:32.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:31.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:32.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:32.072+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.073713+0000) 2024-02-20T09:17:32.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:32.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:32.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:32.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:32.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:32.175+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:32.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:32.176+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:32.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:32.474+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.475802+0000) 2024-02-20T09:17:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:32.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:32.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:32.791+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:32.792+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:32.718+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.719155+0000) 2024-02-20T09:17:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:32.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:32.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:32.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:32.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:32.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:32.988+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.989791+0000) 2024-02-20T09:17:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:33.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:33.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:33.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:33.208+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:33.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:33.477+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:33.478298+0000) 2024-02-20T09:17:33.816 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:33.720+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:33.721671+0000) 2024-02-20T09:17:34.173 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:33.991+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:33.992407+0000) 2024-02-20T09:17:34.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:33.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:34.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:33.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:34.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:33.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:34.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:33.932+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:34.174 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:33.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:34.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:33.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:34.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:33.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:34.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:33.818+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:34.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:34.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:34.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:34.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:34.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:34.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:34.783 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:34.723+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:34.724413+0000) 2024-02-20T09:17:34.783 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:34.480+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:34.481386+0000) 2024-02-20T09:17:35.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:34.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:35.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:34.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:35.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:34.905+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:35.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:34.906+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:35.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:34.994+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:34.995082+0000) 2024-02-20T09:17:35.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:35.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:35.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:35.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:35.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:35.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:35.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:34.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:35.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:34.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:35.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:34.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:35.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:34.850+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:35.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:35.861 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:35.725+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:35.726881+0000) 2024-02-20T09:17:35.861 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:35.482+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:35.483622+0000) 2024-02-20T09:17:36.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:35.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:36.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:35.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:36.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:35.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:36.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:35.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:36.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:35.996+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:35.997576+0000) 2024-02-20T09:17:36.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:36.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:36.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:36.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:36.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:36.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:36.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:35.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:36.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:35.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:36.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:35.885+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:36.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:35.886+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:36.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:36.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:36.484+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.485239+0000) 2024-02-20T09:17:37.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:36.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:37.000 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:36.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:36.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:36.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:36.727+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.728057+0000) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:36.830+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:36.830+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:36.830+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:37.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:36.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:37.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:37.080+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60748 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:37.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:37.077+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.078292+0000) 2024-02-20T09:17:37.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:37.077+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.078293+0000) 2024-02-20T09:17:37.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:36.998+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.999276+0000) 2024-02-20T09:17:37.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:37.079+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.080094+0000) 2024-02-20T09:17:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:37.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:37.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:37.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:37.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:37.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:37.484+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.485420+0000) 2024-02-20T09:17:37.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:37.727+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.728352+0000) 2024-02-20T09:17:37.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:37.871+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:37.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:37.871+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:37.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:37.871+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:38.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:37.998+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.999540+0000) 2024-02-20T09:17:38.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:37.873+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:38.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:37.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:38.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:37.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:38.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:37.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:38.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:37.946+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:38.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:38.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:38.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:38.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:38.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:38.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:38.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:38.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:38.486+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:38.487858+0000) 2024-02-20T09:17:39.003 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:38.729+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:38.730893+0000) 2024-02-20T09:17:39.003 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:38.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:39.003 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:38.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:39.003 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:38.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:39.003 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:38.853+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:39.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:38.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:39.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:38.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:39.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:38.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:39.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:38.975+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:39.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:39.001+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:39.001947+0000) 2024-02-20T09:17:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:39.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:39.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:39.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:39.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:39.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:39.489+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:39.490024+0000) 2024-02-20T09:17:39.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:39.732+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:39.733278+0000) 2024-02-20T09:17:40.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:39.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:39.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:39.879+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:39.880+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:40.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:40.003+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:40.004598+0000) 2024-02-20T09:17:40.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:40.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:40.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:40.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:40.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:40.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:39.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:39.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:39.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:39.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:40.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:40.491+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:40.492751+0000) 2024-02-20T09:17:40.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:40.734+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:40.735714+0000) 2024-02-20T09:17:40.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:40.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:41.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:40.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:41.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:40.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:40.921+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:41.006+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.007192+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:41.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:41.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:41.115+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:41.116+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:40.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:40.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:40.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:40.982+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:41.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:41.493+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.494638+0000) 2024-02-20T09:17:41.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:41.736+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.737062+0000) 2024-02-20T09:17:42.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:42.007+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.008523+0000) 2024-02-20T09:17:42.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:42.085+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.086540+0000) 2024-02-20T09:17:42.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:42.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:42.209 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:42.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:42.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:42.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:42.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:42.154+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:42.210 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:42.087+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60786 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:42.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:41.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:41.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:41.959+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:41.959+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:42.084+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.085353+0000) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:42.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:42.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:42.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:42.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:42.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:42.025+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:42.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:42.084+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.085857+0000) 2024-02-20T09:17:42.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:42.736+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.737245+0000) 2024-02-20T09:17:42.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:42.493+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.494839+0000) 2024-02-20T09:17:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:42.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:42.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:42.951+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:42.952+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:43.007+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:43.008731+0000) 2024-02-20T09:17:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:43.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:43.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:43.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:43.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:43.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:43.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:43.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:43.048+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:43.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:43.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:43.496+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:43.497426+0000) 2024-02-20T09:17:43.817 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:43.738+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:43.739646+0000) 2024-02-20T09:17:44.073 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:44.010+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:44.011158+0000) 2024-02-20T09:17:44.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:43.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:44.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:43.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:44.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:43.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:44.074 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:43.943+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:44.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:44.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:44.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:44.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:44.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:44.171+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:44.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:44.172+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:44.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:44.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:44.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:44.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:44.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:44.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:44.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:44.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:44.499+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:44.499948+0000) 2024-02-20T09:17:45.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:44.741+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:44.741974+0000) 2024-02-20T09:17:45.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:44.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:45.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:44.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:45.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:44.954+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:45.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:44.955+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:45.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:45.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:45.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:45.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:45.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:45.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:45.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:45.080+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:45.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:45.012+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:45.013596+0000) 2024-02-20T09:17:45.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:45.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:45.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:45.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:45.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:45.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:45.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:45.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:45.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:45.501+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:45.502598+0000) 2024-02-20T09:17:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:45.743+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:45.744351+0000) 2024-02-20T09:17:46.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:46.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:46.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:46.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:46.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:46.056+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:46.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:46.057+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:46.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:46.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:46.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:46.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:46.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:46.003+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:46.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:46.004+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:46.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:46.015+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.015972+0000) 2024-02-20T09:17:46.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:46.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:46.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:46.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:46.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:46.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:46.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:46.502+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.503855+0000) 2024-02-20T09:17:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:46.745+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.746126+0000) 2024-02-20T09:17:47.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:47.095+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60827 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:47.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:47.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:47.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.043+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:47.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.044+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:47.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.092+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.093011+0000) 2024-02-20T09:17:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:47.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:47.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:47.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:47.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:47.092+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.093570+0000) 2024-02-20T09:17:47.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:47.016+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.017541+0000) 2024-02-20T09:17:47.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:47.092+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.093748+0000) 2024-02-20T09:17:47.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:47.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:47.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:47.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:47.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:47.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:47.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:47.133+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:47.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:47.503+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.504052+0000) 2024-02-20T09:17:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.745+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.746312+0000) 2024-02-20T09:17:48.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:48.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:48.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:48.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:48.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:48.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:48.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:47.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:48.016+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:48.017731+0000) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:48.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:48.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:48.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:48.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:48.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:48.571 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:48.505+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:48.506116+0000) 2024-02-20T09:17:48.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:48.747+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:48.748885+0000) 2024-02-20T09:17:48.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:48.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:48.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:48.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:48.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:48.956+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:48.957+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:49.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:49.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:49.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:49.047+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:49.019+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:49.020160+0000) 2024-02-20T09:17:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:49.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:49.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:49.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:49.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:49.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:49.507+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:49.508759+0000) 2024-02-20T09:17:49.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:49.749+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:49.750897+0000) 2024-02-20T09:17:49.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:49.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:49.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:49.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:49.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:49.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:49.943+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:50.022+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:50.023629+0000) 2024-02-20T09:17:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:50.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:50.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:50.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:50.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:50.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:50.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:50.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:50.086+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:50.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:50.509+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:50.510794+0000) 2024-02-20T09:17:50.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:50.752+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:50.753723+0000) 2024-02-20T09:17:50.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:50.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:50.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:50.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:50.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:50.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:50.952+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:51.025+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.026032+0000) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:51.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:51.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:51.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:51.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:51.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:51.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:51.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:51.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:51.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:51.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:51.511+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.511978+0000) 2024-02-20T09:17:51.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:51.754+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.754969+0000) 2024-02-20T09:17:52.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:52.102+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60865 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:51.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:51.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:51.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:51.918+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:52.099+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.100693+0000) 2024-02-20T09:17:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:52.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:52.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:52.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:52.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:52.100+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.101232+0000) 2024-02-20T09:17:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:52.026+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.027213+0000) 2024-02-20T09:17:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:52.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:52.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:52.032+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:52.033+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:52.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:52.100+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.101759+0000) 2024-02-20T09:17:52.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:52.511+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.512184+0000) 2024-02-20T09:17:52.881 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:52.754+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.755140+0000) 2024-02-20T09:17:52.881 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:52.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:52.881 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:52.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:53.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:53.026+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:53.027407+0000) 2024-02-20T09:17:53.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:53.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:53.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:53.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:53.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:53.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:53.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:53.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:53.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:52.878+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:53.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:52.879+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:53.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:53.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:53.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:53.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.002+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:53.818 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:53.756+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:53.757628+0000) 2024-02-20T09:17:53.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.514+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:53.515764+0000) 2024-02-20T09:17:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:53.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:53.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:53.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:53.924+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.989+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:54.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:53.990+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:54.028+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:54.029493+0000) 2024-02-20T09:17:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:54.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:54.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:54.053+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:54.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:54.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:54.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:54.516+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:54.517856+0000) 2024-02-20T09:17:54.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:54.759+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:54.759968+0000) 2024-02-20T09:17:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:54.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:54.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:54.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:54.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:54.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:54.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:54.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:54.965+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:55.030+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:55.031928+0000) 2024-02-20T09:17:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:55.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:55.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:55.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:55.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:55.062+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:55.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:55.519+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:55.520426+0000) 2024-02-20T09:17:55.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:55.761+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:55.762766+0000) 2024-02-20T09:17:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:55.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:55.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:55.925+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:56.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:55.926+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:55.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:55.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:55.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:55.887+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:56.033+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.034560+0000) 2024-02-20T09:17:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:56.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:56.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:56.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:56.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:56.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:56.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:56.520+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.521791+0000) 2024-02-20T09:17:56.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:56.763+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.764115+0000) 2024-02-20T09:17:56.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:56.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:56.846 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:56.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:57.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:56.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:57.106 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:56.844+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:57.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:57.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:57.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:57.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:57.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.034+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.035870+0000) 2024-02-20T09:17:57.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:56.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:57.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:56.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:57.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:56.955+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:57.108 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:56.956+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:57.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:17:57.110+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60906 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:17:57.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.107+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.108734+0000) 2024-02-20T09:17:57.369 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:57.108+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.109259+0000) 2024-02-20T09:17:57.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:57.107+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.108008+0000) 2024-02-20T09:17:57.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:57.521+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.521959+0000) 2024-02-20T09:17:58.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:58.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:58.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.989+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:58.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:57.990+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:58.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:57.763+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.764333+0000) 2024-02-20T09:17:58.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:57.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:58.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:57.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:58.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:57.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:58.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:57.819+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:58.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:57.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:58.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:57.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:58.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:57.973+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:58.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:57.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:58.035+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:58.036049+0000) 2024-02-20T09:17:58.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:58.523+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:58.524014+0000) 2024-02-20T09:17:58.845 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:58.766+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:58.767831+0000) 2024-02-20T09:17:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:58.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:58.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:58.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:58.844+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:59.038+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:59.039835+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:59.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:59.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:59.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:17:59.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:17:59.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:59.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:59.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:59.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:17:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:59.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:17:59.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:17:59.525+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:59.526569+0000) 2024-02-20T09:17:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:59.770+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:59.771389+0000) 2024-02-20T09:17:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:59.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:59.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:59.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:17:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:17:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:17:59.819+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:00.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:00.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:00.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:00.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:00.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:00.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:00.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:00.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:00.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:00.041+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:00.041969+0000) 2024-02-20T09:18:00.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:00.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:00.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:00.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:00.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:00.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:00.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:00.070+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:00.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:00.527+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:00.528862+0000) 2024-02-20T09:18:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:00.773+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:00.774775+0000) 2024-02-20T09:18:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:00.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:00.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:00.776+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:00.778+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:01.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:01.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:01.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:01.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:01.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:01.028+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:01.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:01.029+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:01.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:01.043+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.044605+0000) 2024-02-20T09:18:01.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:01.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:01.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:01.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:01.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:01.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:01.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:01.036+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:01.581 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:01.529+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.530020+0000) 2024-02-20T09:18:01.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:01.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:01.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:01.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:01.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:01.757+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:01.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:01.757+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:01.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:01.775+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.775950+0000) 2024-02-20T09:18:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:02.114+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.115735+0000) 2024-02-20T09:18:02.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:02.117+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60944 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:02.044+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.045881+0000) 2024-02-20T09:18:02.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:02.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:02.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:02.068+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:02.069+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:02.117+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.118419+0000) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:02.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:02.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:02.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:02.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:02.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:02.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:02.116+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.117404+0000) 2024-02-20T09:18:02.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:02.529+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.530202+0000) 2024-02-20T09:18:02.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:02.775+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.776139+0000) 2024-02-20T09:18:02.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:02.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:02.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:02.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:02.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:02.780+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:03.046 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:02.781+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:03.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:03.045+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:03.045971+0000) 2024-02-20T09:18:03.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:03.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:03.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:03.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:03.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:03.094+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:03.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:03.095+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:03.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:03.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:03.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:03.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:03.532+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:03.533241+0000) 2024-02-20T09:18:04.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:03.778+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:03.779683+0000) 2024-02-20T09:18:04.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:03.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:04.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:03.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:04.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:03.806+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:04.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:03.807+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:04.047+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:04.048448+0000) 2024-02-20T09:18:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:04.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:04.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:04.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:04.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:04.084+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:04.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:04.085+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:04.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:04.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:04.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:04.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:04.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:04.535+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:04.536866+0000) 2024-02-20T09:18:05.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:04.781+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:04.782126+0000) 2024-02-20T09:18:05.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:04.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:05.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:04.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:05.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:04.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:05.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:04.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:05.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:05.049+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:05.050707+0000) 2024-02-20T09:18:05.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:05.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:05.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:05.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:05.132+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:05.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:05.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:05.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:05.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:05.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:05.538+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:05.539403+0000) 2024-02-20T09:18:06.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:05.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:06.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:05.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:06.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:05.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:06.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:05.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:06.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:05.783+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:05.784508+0000) 2024-02-20T09:18:06.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:06.052+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.053433+0000) 2024-02-20T09:18:06.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:06.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:06.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:06.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:06.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:06.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:06.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:06.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:06.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:06.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:06.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:06.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:06.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:06.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:06.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:06.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:06.540+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.540932+0000) 2024-02-20T09:18:07.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:06.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:07.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:06.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:07.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:06.773+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:07.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:06.774+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:07.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:06.784+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.785616+0000) 2024-02-20T09:18:07.369 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:07.124+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 60985 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:07.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:07.122+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.123557+0000) 2024-02-20T09:18:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:07.121+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.122896+0000) 2024-02-20T09:18:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:07.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:07.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:07.153+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:07.154+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:07.054+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.055470+0000) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:07.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:07.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:07.081+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:07.082+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:07.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:07.122+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.123221+0000) 2024-02-20T09:18:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:07.540+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.541150+0000) 2024-02-20T09:18:08.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:07.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:08.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:07.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:08.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:07.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:08.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:07.766+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:08.057 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:07.784+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.785787+0000) 2024-02-20T09:18:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:08.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:08.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:08.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:08.167+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:08.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:08.054+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:08.055651+0000) 2024-02-20T09:18:08.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:08.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:08.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:08.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:08.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:08.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:08.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:08.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:08.542+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:08.543918+0000) 2024-02-20T09:18:09.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:08.788+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:08.789027+0000) 2024-02-20T09:18:09.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:08.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:09.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:08.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:09.059 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:08.810+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:09.060 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:08.811+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:09.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:09.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:09.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:09.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:09.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:09.175+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:09.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:09.175+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:09.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:09.057+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:09.058367+0000) 2024-02-20T09:18:09.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:09.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:09.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:09.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:09.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:09.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:09.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:09.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:09.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:09.545+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:09.546372+0000) 2024-02-20T09:18:10.061 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:09.770+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:10.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:09.770+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:10.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:09.770+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:10.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:09.771+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:10.062 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:09.790+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:09.791116+0000) 2024-02-20T09:18:10.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:10.059+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:10.060526+0000) 2024-02-20T09:18:10.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:10.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:10.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:10.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:10.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:10.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:10.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:10.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:10.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:10.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:10.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:10.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:10.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:10.208+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:10.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:10.209+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:10.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:10.547+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:10.548740+0000) 2024-02-20T09:18:11.064 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:10.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:11.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:10.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:11.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:10.789+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:11.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:10.790+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:11.065 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:10.793+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:10.793954+0000) 2024-02-20T09:18:11.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:11.062+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.063869+0000) 2024-02-20T09:18:11.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:11.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:11.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:11.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:11.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:11.120+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:11.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:11.121+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:11.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:11.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:11.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:11.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:11.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:11.249+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:11.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:11.250+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:11.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:11.549+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.550338+0000) 2024-02-20T09:18:11.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:11.794+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.795126+0000) 2024-02-20T09:18:11.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:11.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:11.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:11.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:11.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:11.823+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:11.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:11.825+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:12.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:12.131+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61024 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:12.128+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.129392+0000) 2024-02-20T09:18:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:12.128+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.129542+0000) 2024-02-20T09:18:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:12.064+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.065705+0000) 2024-02-20T09:18:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:12.128+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.129587+0000) 2024-02-20T09:18:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:12.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:12.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:12.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:12.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:12.144+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:12.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:12.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:12.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:12.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:12.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:12.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:12.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:12.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:12.549+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.550515+0000) 2024-02-20T09:18:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:12.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:12.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:12.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:12.778+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:12.794+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.795222+0000) 2024-02-20T09:18:13.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:13.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:13.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:13.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:13.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:13.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:13.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:13.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:13.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:13.064+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:13.065920+0000) 2024-02-20T09:18:13.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:13.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:13.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:13.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:13.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:13.132+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:13.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:13.133+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:13.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:13.553+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:13.554078+0000) 2024-02-20T09:18:14.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:13.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:14.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:13.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:14.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:13.754+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:14.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:13.755+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:14.071 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:13.797+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:13.798773+0000) 2024-02-20T09:18:14.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:14.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:14.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:14.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:14.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:14.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:14.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:14.283+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:14.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:14.068+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:14.069474+0000) 2024-02-20T09:18:14.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:14.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:14.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:14.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:14.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:14.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:14.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:14.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:14.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:14.556+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:14.557703+0000) 2024-02-20T09:18:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:14.800+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:14.801348+0000) 2024-02-20T09:18:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:14.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:14.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:14.805+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:14.805+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:15.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:15.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:15.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:15.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:15.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:15.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:15.071+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:15.072929+0000) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:15.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:15.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:15.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:15.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:15.077+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:15.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:15.559+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:15.560297+0000) 2024-02-20T09:18:16.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:15.803+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:15.804761+0000) 2024-02-20T09:18:16.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:15.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:16.077 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:15.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:16.077 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:15.852+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:16.077 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:15.853+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:16.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:16.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:16.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:16.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:16.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:16.248+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:16.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:16.074+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.075627+0000) 2024-02-20T09:18:16.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:16.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:16.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:16.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:16.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:16.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:16.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:16.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:16.561+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.562388+0000) 2024-02-20T09:18:17.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:16.805+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.806643+0000) 2024-02-20T09:18:17.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:16.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:17.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:16.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:17.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:16.895+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:17.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:16.896+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:17.370 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:17.139+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61064 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:17.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:17.137+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.138059+0000) 2024-02-20T09:18:17.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:17.136+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.137344+0000) 2024-02-20T09:18:17.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:17.206+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:17.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:17.206+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:17.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:17.206+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:17.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:17.206+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:17.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:17.076+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.077619+0000) 2024-02-20T09:18:17.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:17.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:17.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:17.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:17.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:17.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:17.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:17.090+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:17.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:17.136+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.137354+0000) 2024-02-20T09:18:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:17.561+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.562593+0000) 2024-02-20T09:18:18.078 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:17.805+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.806823+0000) 2024-02-20T09:18:18.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:17.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:18.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:17.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:18.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:17.931+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:18.079 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:17.932+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:18.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:18.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:18.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:18.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:18.231+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:18.076+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:18.077831+0000) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:18.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:18.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:18.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:18.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:18.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:18.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:18.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:18.564+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:18.565082+0000) 2024-02-20T09:18:19.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:18.809+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:18.810424+0000) 2024-02-20T09:18:19.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:18.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:19.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:18.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:19.081 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:18.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:19.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:18.950+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:19.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:19.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:19.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:19.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:19.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:19.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:19.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:19.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:19.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:19.079+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:19.080196+0000) 2024-02-20T09:18:19.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:19.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:19.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:19.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:19.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:19.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:19.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:19.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:19.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:19.566+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:19.567534+0000) 2024-02-20T09:18:20.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:19.812+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:19.812965+0000) 2024-02-20T09:18:20.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:19.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:20.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:19.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:20.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:19.997+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:20.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:19.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:20.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:20.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:20.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:20.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:20.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:20.277+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:20.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:20.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:20.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:20.081+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:20.082583+0000) 2024-02-20T09:18:20.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:20.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:20.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:20.151+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:20.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:20.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:20.567+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:20.569761+0000) 2024-02-20T09:18:21.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:20.813+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:20.815810+0000) 2024-02-20T09:18:21.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:20.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:21.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:20.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:21.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:20.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:21.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:20.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:21.083+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.085642+0000) 2024-02-20T09:18:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:21.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:21.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:21.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:21.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:21.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:21.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:21.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:21.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:21.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:21.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:21.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:21.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:21.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:21.262+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:21.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:21.569+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.571415+0000) 2024-02-20T09:18:22.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:21.815+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.817564+0000) 2024-02-20T09:18:22.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:22.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:22.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:22.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:22.091 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:22.028+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:22.092 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:22.029+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:22.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:22.145+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61103 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:22.270 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:22.142+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.144259+0000) 2024-02-20T09:18:22.271 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:22.142+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.144801+0000) 2024-02-20T09:18:22.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:22.084+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.086756+0000) 2024-02-20T09:18:22.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:22.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:22.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:22.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:22.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:22.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:22.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:22.135+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:22.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:22.143+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.145802+0000) 2024-02-20T09:18:22.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:22.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:22.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:22.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:22.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:22.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:22.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:22.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:22.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:22.569+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.571596+0000) 2024-02-20T09:18:23.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:22.815+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.817729+0000) 2024-02-20T09:18:23.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:23.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:23.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.009+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:23.088 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:23.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:23.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:23.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:23.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:23.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:23.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:23.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:23.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:23.085+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:23.086975+0000) 2024-02-20T09:18:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:23.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:23.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:23.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:23.112+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:23.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:23.573+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:23.574883+0000) 2024-02-20T09:18:24.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.819+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:23.820269+0000) 2024-02-20T09:18:24.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:24.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:24.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:24.090 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:23.976+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:24.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:24.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:24.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:24.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:24.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:24.087+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:24.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:24.088+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:24.089343+0000) 2024-02-20T09:18:24.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:24.088+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:24.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:24.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:24.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:24.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:24.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:24.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:24.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:24.288+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:24.576+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:24.577326+0000) 2024-02-20T09:18:25.075 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:25.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:25.075 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:25.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:25.075 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:25.071+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:25.075 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:25.072+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:25.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:24.822+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:24.823795+0000) 2024-02-20T09:18:25.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:25.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:25.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:25.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:25.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:25.090+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:25.091700+0000) 2024-02-20T09:18:25.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:25.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:25.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:25.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:25.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:25.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:25.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:25.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:25.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:25.578+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:25.579913+0000) 2024-02-20T09:18:26.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.825+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:25.826261+0000) 2024-02-20T09:18:26.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:26.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:26.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:26.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:25.999+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:26.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:26.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:26.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:26.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:26.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:26.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:26.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:26.291+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:26.093+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.094195+0000) 2024-02-20T09:18:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:26.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:26.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:26.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:26.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:26.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:26.580+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.581168+0000) 2024-02-20T09:18:27.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:26.826+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.827882+0000) 2024-02-20T09:18:27.097 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:27.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:27.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:27.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:27.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:27.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:27.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:27.024+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:27.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:27.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:27.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:27.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:27.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:27.063+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:27.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:27.063+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:27.099 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:27.095+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.096174+0000) 2024-02-20T09:18:27.371 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:27.154+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61143 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:27.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:27.152+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.153552+0000) 2024-02-20T09:18:27.372 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:27.150+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.151123+0000) 2024-02-20T09:18:27.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:27.150+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.151677+0000) 2024-02-20T09:18:27.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:27.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:27.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:27.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:27.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:27.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:27.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:27.269+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:27.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:27.580+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.581694+0000) 2024-02-20T09:18:28.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:28.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:28.097 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:28.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:28.023+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:28.024+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:27.827+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.828102+0000) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:28.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:28.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:28.044+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:28.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:28.045+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:28.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:28.095+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:28.096353+0000) 2024-02-20T09:18:28.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:28.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:28.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:28.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:28.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:28.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:28.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:28.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:28.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:28.583+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:28.584218+0000) 2024-02-20T09:18:29.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:29.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:29.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:29.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:29.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:29.045+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:29.083 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:29.046+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:29.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:28.830+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:28.831663+0000) 2024-02-20T09:18:29.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:29.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:29.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:29.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:29.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:29.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:29.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:29.081+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:29.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:29.097+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:29.098844+0000) 2024-02-20T09:18:29.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:29.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:29.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:29.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:29.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:29.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:29.585+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:29.586691+0000) 2024-02-20T09:18:30.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:29.834+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:29.835198+0000) 2024-02-20T09:18:30.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:30.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:30.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:30.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:30.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:30.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:30.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:30.064+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:30.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:30.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:30.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:30.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:30.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:30.313+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:30.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:30.314+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:30.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:30.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:30.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:30.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:30.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:30.092+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:30.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:30.093+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:30.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:30.100+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:30.101050+0000) 2024-02-20T09:18:30.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:30.587+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:30.588837+0000) 2024-02-20T09:18:31.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:30.836+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:30.837906+0000) 2024-02-20T09:18:31.105 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:31.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:31.105 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:31.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:31.105 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:31.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:31.105 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:31.061+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:31.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:31.102+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.103705+0000) 2024-02-20T09:18:31.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:31.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:31.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:31.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:31.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:31.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:31.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:31.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:31.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:31.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:31.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:31.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:31.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:31.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:31.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:31.356+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:31.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:31.589+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.590849+0000) 2024-02-20T09:18:31.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:31.838+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.839179+0000) 2024-02-20T09:18:32.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:32.158+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.159268+0000) 2024-02-20T09:18:32.222 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:32.162+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61182 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:32.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:32.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:32.080+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:32.081+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:32.158+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.159359+0000) 2024-02-20T09:18:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:32.104+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.105197+0000) 2024-02-20T09:18:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:32.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:32.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:32.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:32.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:32.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:32.158+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.159913+0000) 2024-02-20T09:18:32.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:32.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:32.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:32.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:32.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:32.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:32.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:32.395+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:32.838+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.839402+0000) 2024-02-20T09:18:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:32.590+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.591028+0000) 2024-02-20T09:18:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:33.104+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:33.105399+0000) 2024-02-20T09:18:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:33.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:33.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:33.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:33.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:33.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:33.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:33.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:33.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:33.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:33.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:33.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:33.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:33.046+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:33.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:33.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:33.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:33.391+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:33.392+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:33.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:33.592+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:33.592976+0000) 2024-02-20T09:18:34.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:33.841+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:33.841985+0000) 2024-02-20T09:18:34.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:34.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:34.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:34.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:34.108 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:34.045+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:34.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:34.046+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:34.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:34.106+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:34.107343+0000) 2024-02-20T09:18:34.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:34.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:34.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:34.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:34.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:34.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:34.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:34.163+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:34.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:34.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:34.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:34.437+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:34.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:34.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:34.595+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:34.596626+0000) 2024-02-20T09:18:35.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:34.844+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:34.845589+0000) 2024-02-20T09:18:35.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:35.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:35.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:35.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:35.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:35.065+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:35.111 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:35.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:35.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:35.109+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:35.110160+0000) 2024-02-20T09:18:35.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:35.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:35.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:35.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:35.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:35.197+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:35.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:35.198+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:35.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:35.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:35.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:35.483+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:35.483+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:35.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:35.598+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:35.599048+0000) 2024-02-20T09:18:36.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:35.847+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:35.848118+0000) 2024-02-20T09:18:36.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:36.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:36.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:36.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:36.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:36.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:36.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:36.057+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:36.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:36.112+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.113707+0000) 2024-02-20T09:18:36.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:36.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:36.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:36.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:36.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:36.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:36.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:36.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:36.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:36.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:36.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:36.489+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:36.600+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.601139+0000) 2024-02-20T09:18:37.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:36.849+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.850233+0000) 2024-02-20T09:18:37.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:37.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:37.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:37.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:37.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:37.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:37.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:37.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:37.372 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:37.166+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.167825+0000) 2024-02-20T09:18:37.373 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:37.169+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61222 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:37.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:37.167+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.168916+0000) 2024-02-20T09:18:37.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:37.114+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.115070+0000) 2024-02-20T09:18:37.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:37.167+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.168407+0000) 2024-02-20T09:18:37.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:37.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:37.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:37.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:37.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:37.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:37.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:37.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:37.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:37.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:37.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:37.470+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:37.600+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.601317+0000) 2024-02-20T09:18:38.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:37.849+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.850447+0000) 2024-02-20T09:18:38.372 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:38.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:38.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:38.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:38.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:38.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:38.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:38.102+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:38.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:38.114+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:38.115239+0000) 2024-02-20T09:18:38.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:38.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:38.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:38.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:38.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:38.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:38.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:38.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:38.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:38.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:38.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:38.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:38.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:38.603+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:38.604877+0000) 2024-02-20T09:18:39.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:38.853+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:38.853950+0000) 2024-02-20T09:18:39.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:39.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:39.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:39.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:39.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:39.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:39.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:39.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:39.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:39.116+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:39.117622+0000) 2024-02-20T09:18:39.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:39.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:39.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:39.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:39.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:39.263+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:39.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:39.264+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:39.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:39.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:39.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:39.537+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:39.538+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:39.606+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:39.607363+0000) 2024-02-20T09:18:40.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:39.854+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:39.855571+0000) 2024-02-20T09:18:40.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:40.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:40.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:40.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:40.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:40.127+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:40.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:40.128+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:40.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:40.118+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:40.119811+0000) 2024-02-20T09:18:40.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:40.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:40.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:40.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:40.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:40.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:40.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:40.242+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:40.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:40.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:40.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:40.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:40.581+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:40.610+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:40.610951+0000) 2024-02-20T09:18:41.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:40.857+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:40.858030+0000) 2024-02-20T09:18:41.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:41.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:41.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:41.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:41.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:41.173+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:41.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:41.173+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:41.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:41.121+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.122396+0000) 2024-02-20T09:18:41.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:41.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:41.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:41.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:41.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:41.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:41.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:41.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:41.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:41.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:41.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:41.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:41.608+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:41.611+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.612216+0000) 2024-02-20T09:18:42.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:41.858+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.859658+0000) 2024-02-20T09:18:42.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:42.177+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61261 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:42.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:42.174+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.175229+0000) 2024-02-20T09:18:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:42.123+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.124488+0000) 2024-02-20T09:18:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:42.174+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.175368+0000) 2024-02-20T09:18:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:42.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:42.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:42.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:42.301+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:42.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:42.302+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:42.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:42.174+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.175091+0000) 2024-02-20T09:18:42.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:42.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:42.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:42.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:42.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:42.176+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:42.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:42.178+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:42.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:42.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:42.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:42.587+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:42.588+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:42.611+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.612411+0000) 2024-02-20T09:18:43.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:42.858+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.859830+0000) 2024-02-20T09:18:43.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:43.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:43.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:43.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:43.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:43.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:43.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:43.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:43.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:43.123+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:43.124694+0000) 2024-02-20T09:18:43.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:43.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:43.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:43.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:43.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:43.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:43.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:43.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:43.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:43.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:43.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:43.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:43.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:43.559+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:43.614+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:43.615097+0000) 2024-02-20T09:18:44.128 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:43.862+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:43.863408+0000) 2024-02-20T09:18:44.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:44.126+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:44.127152+0000) 2024-02-20T09:18:44.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:44.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:44.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:44.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:44.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:44.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:44.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:44.192+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:44.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:44.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:44.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:44.524+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:44.616+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:44.617358+0000) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:44.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:44.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:44.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:44.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:44.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:45.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:44.864+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:44.865474+0000) 2024-02-20T09:18:45.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:45.129+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:45.130424+0000) 2024-02-20T09:18:45.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:45.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:45.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:45.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:45.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:45.149+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:45.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:45.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:45.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:45.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:45.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:45.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:45.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:45.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:45.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:45.477+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:45.618+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:45.619433+0000) 2024-02-20T09:18:46.134 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:45.868+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:45.869052+0000) 2024-02-20T09:18:46.134 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:46.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:46.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:46.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:46.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:46.132+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:46.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:46.133+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.133956+0000) 2024-02-20T09:18:46.392 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:46.133+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:46.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:46.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:46.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:46.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:46.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:46.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:46.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:46.445+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:46.620+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.621047+0000) 2024-02-20T09:18:47.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:46.869+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.870281+0000) 2024-02-20T09:18:47.398 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:47.184+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61301 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:47.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:47.183+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.184153+0000) 2024-02-20T09:18:47.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:47.134+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.135473+0000) 2024-02-20T09:18:47.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:47.182+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.183108+0000) 2024-02-20T09:18:47.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:47.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:47.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:47.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:47.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:47.146+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:47.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:47.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:47.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:47.182+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.183184+0000) 2024-02-20T09:18:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:47.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:47.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:47.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:47.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:47.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:47.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:47.446+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:47.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:47.620+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.621252+0000) 2024-02-20T09:18:48.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:47.869+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.870479+0000) 2024-02-20T09:18:48.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:48.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:48.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:48.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:48.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:48.113+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:48.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:48.114+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:48.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:48.134+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:48.135662+0000) 2024-02-20T09:18:48.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:48.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:48.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:48.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:48.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:48.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:48.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:48.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:48.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:48.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:48.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:48.410+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:48.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:48.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:48.623+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:48.624414+0000) 2024-02-20T09:18:49.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:48.871+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:48.872631+0000) 2024-02-20T09:18:49.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:49.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:49.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:49.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:49.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:49.082+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:49.139 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:49.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:49.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:49.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:49.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:49.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:49.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:49.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:49.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:49.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:49.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:49.137+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:49.138262+0000) 2024-02-20T09:18:49.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:49.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:49.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:49.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:49.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:49.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:49.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:49.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:49.626+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:49.627852+0000) 2024-02-20T09:18:50.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:49.874+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:49.875217+0000) 2024-02-20T09:18:50.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:50.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:50.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:50.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:50.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:50.073+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:50.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:50.074+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:50.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:50.139+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:50.140459+0000) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:50.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:50.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:50.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:50.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:50.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:50.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:50.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:50.350+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:50.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:50.351+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:50.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:50.630+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:50.631153+0000) 2024-02-20T09:18:51.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:50.876+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:50.877897+0000) 2024-02-20T09:18:51.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:51.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:51.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:51.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:51.144 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:51.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:51.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:51.036+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:51.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:51.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:51.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:51.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:51.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:51.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:51.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:51.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:51.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:51.142+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.143368+0000) 2024-02-20T09:18:51.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:51.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:51.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:51.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:51.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:51.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:51.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:51.371+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:51.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:51.632+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.633288+0000) 2024-02-20T09:18:52.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:51.878+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.879390+0000) 2024-02-20T09:18:52.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:52.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:52.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:52.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:52.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:52.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:52.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:52.081+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:52.399 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:52.192+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61340 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:52.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:52.189+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.190656+0000) 2024-02-20T09:18:52.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:52.189+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.190480+0000) 2024-02-20T09:18:52.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:52.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:52.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:52.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:52.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:52.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:52.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:52.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:52.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:52.144+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.145174+0000) 2024-02-20T09:18:52.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:52.189+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.190818+0000) 2024-02-20T09:18:52.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:52.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:52.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:52.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:52.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:52.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:52.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:52.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:52.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:52.632+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.633497+0000) 2024-02-20T09:18:53.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:52.878+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.879569+0000) 2024-02-20T09:18:53.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:53.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:53.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:53.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:53.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:53.071+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:53.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:53.072+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:53.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:53.144+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:53.145435+0000) 2024-02-20T09:18:53.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:53.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:53.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:53.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:53.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:53.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:53.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:53.317+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:53.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:53.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:53.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:53.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:53.415+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:53.634+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:53.635639+0000) 2024-02-20T09:18:54.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:53.881+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:53.882844+0000) 2024-02-20T09:18:54.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:54.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:54.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:54.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:54.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:54.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:54.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:54.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:54.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:54.148+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:54.148973+0000) 2024-02-20T09:18:54.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:54.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:54.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:54.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:54.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:54.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:54.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:54.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:54.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:54.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:54.439+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:54.440+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:54.637+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:54.638221+0000) 2024-02-20T09:18:55.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:54.883+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:54.884837+0000) 2024-02-20T09:18:55.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:55.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:55.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.026+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:55.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.027+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:55.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:55.150+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:55.151414+0000) 2024-02-20T09:18:55.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:55.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:55.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:55.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:55.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:55.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:55.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:55.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:55.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:55.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:55.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:55.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:55.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:55.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:55.639+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:55.640923+0000) 2024-02-20T09:18:56.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.887+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:55.888181+0000) 2024-02-20T09:18:56.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:56.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:56.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:56.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:55.984+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:56.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:56.153+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.154061+0000) 2024-02-20T09:18:56.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:56.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:56.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:56.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:56.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:56.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:56.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:56.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:56.465+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:56.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:56.466+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:56.467+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:56.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.642953+0000) 2024-02-20T09:18:57.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:56.889+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.890275+0000) 2024-02-20T09:18:57.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:56.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:57.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:56.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:57.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:56.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:57.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:56.984+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:57.429 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:18:57.200+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61380 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:18:57.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:57.196+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.197906+0000) 2024-02-20T09:18:57.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:57.196+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.197897+0000) 2024-02-20T09:18:57.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:57.155+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.156138+0000) 2024-02-20T09:18:57.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:57.198+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.199322+0000) 2024-02-20T09:18:57.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:57.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:57.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:57.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:57.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:57.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:57.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:57.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:57.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:57.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:57.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:57.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:57.477+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:57.478+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:57.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:57.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.643126+0000) 2024-02-20T09:18:58.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:57.889+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.890463+0000) 2024-02-20T09:18:58.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:57.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:58.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:57.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:58.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:57.962+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:58.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:57.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:58.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:58.155+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:58.156320+0000) 2024-02-20T09:18:58.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:58.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:58.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:58.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:58.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:58.287+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:58.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:58.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:58.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:58.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:58.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:58.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:58.440+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:58.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:58.644+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:58.645104+0000) 2024-02-20T09:18:59.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:58.893+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:58.893966+0000) 2024-02-20T09:18:59.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:58.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:59.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:58.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:59.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:58.992+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:18:59.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:58.993+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:59.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:59.157+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:59.158901+0000) 2024-02-20T09:18:59.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:59.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:59.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:59.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:59.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:59.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:18:59.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:18:59.329+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:59.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:59.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:59.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:18:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:59.415+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:18:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:18:59.646+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:59.647739+0000) 2024-02-20T09:19:00.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:59.896+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:59.897361+0000) 2024-02-20T09:19:00.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:59.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:00.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:59.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:00.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:59.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:00.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:18:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:18:59.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:00.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:00.161+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:00.162264+0000) 2024-02-20T09:19:00.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:00.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:00.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:00.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:00.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:00.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:00.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:00.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:00.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:00.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:00.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:00.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:00.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:00.650+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:00.651378+0000) 2024-02-20T09:19:01.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:00.898+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:00.899353+0000) 2024-02-20T09:19:01.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:01.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:01.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:01.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.015+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:01.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:01.163+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.164727+0000) 2024-02-20T09:19:01.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:01.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:01.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:01.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:01.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:01.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:01.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:01.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:01.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:01.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:01.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:01.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:01.652+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.653059+0000) 2024-02-20T09:19:02.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.900+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.901559+0000) 2024-02-20T09:19:02.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:02.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:02.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:02.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:01.973+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:02.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:02.207+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61419 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:02.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:02.205+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.206482+0000) 2024-02-20T09:19:02.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:02.204+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.205840+0000) 2024-02-20T09:19:02.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:02.165+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.166839+0000) 2024-02-20T09:19:02.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:02.205+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.206244+0000) 2024-02-20T09:19:02.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:02.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:02.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:02.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:02.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:02.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:02.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:02.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:02.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:02.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:02.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:02.447+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:02.447+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:02.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:02.652+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.653253+0000) 2024-02-20T09:19:03.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:02.900+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.901737+0000) 2024-02-20T09:19:03.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:03.016+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:03.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:03.016+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:03.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:03.016+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:03.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:03.018+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:03.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:03.166+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:03.166959+0000) 2024-02-20T09:19:03.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:03.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:03.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:03.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:03.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:03.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:03.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:03.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:03.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:03.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:03.402+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:03.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:03.403+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:03.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:03.655+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:03.656520+0000) 2024-02-20T09:19:04.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:03.902+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:03.903920+0000) 2024-02-20T09:19:04.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:04.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:04.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:04.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:04.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:04.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:04.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:04.043+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:04.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:04.168+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:04.169100+0000) 2024-02-20T09:19:04.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:04.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:04.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:04.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:04.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:04.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:04.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:04.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:04.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:04.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:04.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:04.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:04.658+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:04.659093+0000) 2024-02-20T09:19:05.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:04.905+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:04.906642+0000) 2024-02-20T09:19:05.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:05.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:05.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:05.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:05.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:05.025+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:05.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:05.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:05.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:05.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:05.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:05.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:05.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:05.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:05.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:05.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:05.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:05.170+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:05.171390+0000) 2024-02-20T09:19:05.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:05.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:05.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:05.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:05.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:05.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:05.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:05.324+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:05.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:05.660+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:05.661587+0000) 2024-02-20T09:19:06.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:05.909+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:05.910098+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:06.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:06.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:06.053+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:06.054+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:06.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:06.173+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.174516+0000) 2024-02-20T09:19:06.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:06.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:06.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:06.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:06.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:06.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:06.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:06.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:06.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:06.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:06.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:06.435+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:06.436+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:06.662+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.663327+0000) 2024-02-20T09:19:07.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:06.911+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.912240+0000) 2024-02-20T09:19:07.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:07.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:07.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:07.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:07.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:07.081+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:07.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:07.082+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:07.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:07.216+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61459 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:07.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:07.212+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.213018+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:07.211+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.212816+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:07.175+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.176795+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:07.212+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.213754+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:07.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:07.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:07.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:07.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:07.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:07.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:07.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:07.440+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:07.441+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:07.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:07.662+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.663505+0000) 2024-02-20T09:19:08.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:07.911+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.912447+0000) 2024-02-20T09:19:08.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:08.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:08.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:08.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:08.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:08.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:08.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:08.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:08.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:08.176+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:08.177051+0000) 2024-02-20T09:19:08.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:08.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:08.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:08.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:08.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:08.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:08.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:08.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:08.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:08.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:08.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:08.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:08.487+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:08.664+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:08.665899+0000) 2024-02-20T09:19:09.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:08.914+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:08.914947+0000) 2024-02-20T09:19:09.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:09.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:09.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:09.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:09.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:09.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:09.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:09.152+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:09.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:09.178+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:09.179604+0000) 2024-02-20T09:19:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:09.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:09.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:09.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:09.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:09.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:09.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:09.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:09.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:09.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:09.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:09.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:09.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:09.482+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:09.920 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:09.668+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:09.669401+0000) 2024-02-20T09:19:10.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:09.917+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:09.918489+0000) 2024-02-20T09:19:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:10.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:10.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:10.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:10.190+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:10.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:10.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:10.181+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:10.182293+0000) 2024-02-20T09:19:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:10.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:10.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:10.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:10.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:10.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:10.370+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:10.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:10.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:10.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:10.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:10.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:10.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:10.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:10.495+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:10.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:10.670+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:10.671853+0000) 2024-02-20T09:19:11.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:10.920+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:10.921451+0000) 2024-02-20T09:19:11.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:11.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:11.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:11.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:11.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:11.184+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.184955+0000) 2024-02-20T09:19:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:11.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:11.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:11.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:11.367+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:11.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:11.368+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:11.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:11.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:11.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:11.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:11.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:11.514+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:11.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:11.515+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:11.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:11.672+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.673350+0000) 2024-02-20T09:19:12.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:11.922+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.923435+0000) 2024-02-20T09:19:12.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:12.220+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.221048+0000) 2024-02-20T09:19:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:12.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:12.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:12.192+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:12.193+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:12.221+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.221948+0000) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:12.185+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.186384+0000) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:12.221+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.222206+0000) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:12.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:12.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:12.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:12.412+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:12.473 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:12.224+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61498 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:12.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:12.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:12.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:12.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:12.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:12.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:12.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:12.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:12.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:12.672+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.673531+0000) 2024-02-20T09:19:13.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:12.922+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.923614+0000) 2024-02-20T09:19:13.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:13.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:13.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:13.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:13.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:13.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:13.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:13.152+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:13.184+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:13.186697+0000) 2024-02-20T09:19:13.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:13.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:13.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:13.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:13.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:13.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:13.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:13.574+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:13.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:13.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:13.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:13.575+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:13.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:13.575+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:13.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:13.673+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:13.675905+0000) 2024-02-20T09:19:14.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:13.923+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:13.925906+0000) 2024-02-20T09:19:14.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:14.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:14.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:14.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:14.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:14.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:14.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:14.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:14.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:14.187+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:14.189047+0000) 2024-02-20T09:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:14.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:14.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:14.327+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:14.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:14.328+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:14.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:14.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:14.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:14.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:14.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:14.559+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:14.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:14.560+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:14.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:14.676+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:14.678501+0000) 2024-02-20T09:19:15.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:14.926+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:14.928299+0000) 2024-02-20T09:19:15.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:15.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:15.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:15.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:15.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:15.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:15.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:15.131+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:15.189+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:15.191638+0000) 2024-02-20T09:19:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:15.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:15.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:15.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:15.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:15.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:15.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:15.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:15.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:15.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:15.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:15.525+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:15.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:15.525+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:15.933 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:15.679+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:15.681215+0000) 2024-02-20T09:19:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:15.929+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:15.931485+0000) 2024-02-20T09:19:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:16.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:16.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:16.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:16.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:16.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:16.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:16.194+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.195093+0000) 2024-02-20T09:19:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:16.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:16.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:16.338+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:16.339+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:16.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:16.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:16.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:16.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:16.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:16.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:16.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:16.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:16.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:16.681+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.682528+0000) 2024-02-20T09:19:17.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:16.932+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.933446+0000) 2024-02-20T09:19:17.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:17.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:17.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:17.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:17.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:17.151+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:17.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:17.152+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:17.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:17.228+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.229804+0000) 2024-02-20T09:19:17.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:17.231+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61537 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:17.195+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.196843+0000) 2024-02-20T09:19:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:17.228+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.229265+0000) 2024-02-20T09:19:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:17.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:17.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:17.306+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:17.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:17.307+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:17.228+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.229855+0000) 2024-02-20T09:19:17.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:17.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:17.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:17.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:17.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:17.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:17.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:17.541+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:17.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:17.681+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.682736+0000) 2024-02-20T09:19:18.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:17.932+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.933747+0000) 2024-02-20T09:19:18.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:18.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:18.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:18.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:18.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:18.143+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:18.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:18.144+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:18.196+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:18.197065+0000) 2024-02-20T09:19:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:18.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:18.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:18.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:18.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:18.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:18.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:18.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:18.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:18.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:18.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:18.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:18.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:18.537+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:18.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:18.684+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:18.685238+0000) 2024-02-20T09:19:19.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:18.935+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:18.936446+0000) 2024-02-20T09:19:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:19.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:19.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:19.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:19.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:19.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:19.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:19.198+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:19.199556+0000) 2024-02-20T09:19:19.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:19.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:19.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:19.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:19.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:19.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:19.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:19.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:19.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:19.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:19.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:19.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:19.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:19.496+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:19.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:19.686+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:19.687918+0000) 2024-02-20T09:19:20.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:19.938+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:19.939122+0000) 2024-02-20T09:19:20.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:20.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:20.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:20.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:20.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:20.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:20.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:20.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:20.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:20.201+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:20.201981+0000) 2024-02-20T09:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:20.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:20.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:20.309+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:20.310+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:20.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:20.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:20.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:20.468+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:20.469+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:20.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:20.690+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:20.691083+0000) 2024-02-20T09:19:21.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:20.940+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:20.941706+0000) 2024-02-20T09:19:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:21.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:21.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:21.214+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:21.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:21.215+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:21.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:21.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:21.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:21.432+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:21.203+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.204573+0000) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:21.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:21.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:21.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:21.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:21.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:21.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:21.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:21.691+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.692655+0000) 2024-02-20T09:19:22.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:21.942+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.943354+0000) 2024-02-20T09:19:22.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:22.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:22.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:22.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:22.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:22.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:22.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:22.188+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:22.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:22.238+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61577 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:22.235+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.236240+0000) 2024-02-20T09:19:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:22.235+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.236873+0000) 2024-02-20T09:19:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:22.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:22.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:22.388+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:22.389+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:22.205+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.206362+0000) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:22.235+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.236247+0000) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:22.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:22.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:22.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:22.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:22.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:22.691+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.692837+0000) 2024-02-20T09:19:23.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:22.942+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.943561+0000) 2024-02-20T09:19:23.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:23.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:23.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:23.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:23.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:23.433+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:23.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:23.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:23.228+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:23.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:23.229+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:23.205+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:23.206528+0000) 2024-02-20T09:19:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:23.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:23.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:23.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:23.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:23.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:23.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:23.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:23.694+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:23.695078+0000) 2024-02-20T09:19:24.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:23.946+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:23.947114+0000) 2024-02-20T09:19:24.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:24.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:24.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:24.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:24.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:24.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:24.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:24.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:24.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:24.208+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:24.208975+0000) 2024-02-20T09:19:24.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:24.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:24.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:24.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:24.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:24.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:24.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:24.295+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:24.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:24.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:24.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:24.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:24.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:24.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:24.461+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:24.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:24.697+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:24.698495+0000) 2024-02-20T09:19:25.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:24.949+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:24.950556+0000) 2024-02-20T09:19:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:25.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:25.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:25.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:25.306+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:25.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:25.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:25.210+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:25.211227+0000) 2024-02-20T09:19:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:25.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:25.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:25.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:25.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:25.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:25.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:25.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:25.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:25.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:25.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:25.481+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:25.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:25.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:25.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:25.701+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:25.702092+0000) 2024-02-20T09:19:26.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:25.953+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:25.954104+0000) 2024-02-20T09:19:26.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:26.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:26.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:26.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:26.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:26.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:26.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:26.213+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.214106+0000) 2024-02-20T09:19:26.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:26.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:26.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:26.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:26.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:26.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:26.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:26.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:26.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:26.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:26.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:26.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:26.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:26.300+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:26.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:26.301+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:26.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:26.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:26.702+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.703437+0000) 2024-02-20T09:19:27.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:26.954+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.955824+0000) 2024-02-20T09:19:27.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:27.248+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61616 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:27.243+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.244526+0000) 2024-02-20T09:19:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:27.243+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.244137+0000) 2024-02-20T09:19:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:27.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:27.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:27.273+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:27.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:27.273+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:27.215+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.215943+0000) 2024-02-20T09:19:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:27.243+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.244732+0000) 2024-02-20T09:19:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:27.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:27.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:27.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:27.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:27.301+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:27.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:27.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:27.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:27.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:27.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:27.480+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:27.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:27.481+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:27.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:27.702+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.703665+0000) 2024-02-20T09:19:28.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:27.955+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.956034+0000) 2024-02-20T09:19:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:28.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:28.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:28.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:28.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:28.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:28.215+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:28.216145+0000) 2024-02-20T09:19:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:28.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:28.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:28.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:28.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:28.278+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:28.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:28.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:28.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:28.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:28.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:28.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:28.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:28.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:28.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:28.705+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:28.706188+0000) 2024-02-20T09:19:29.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:28.957+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:28.958615+0000) 2024-02-20T09:19:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:29.218+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:29.219026+0000) 2024-02-20T09:19:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:29.268+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:29.268+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:29.268+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:29.269+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:29.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:29.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:29.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:29.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:29.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:29.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:29.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:29.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:29.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:29.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:29.503+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:29.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:29.504+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:29.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:29.707+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:29.708242+0000) 2024-02-20T09:19:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:29.960+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:29.961315+0000) 2024-02-20T09:19:30.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:30.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:30.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:30.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:30.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:30.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:30.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:30.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:30.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:30.220+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:30.221892+0000) 2024-02-20T09:19:30.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:30.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:30.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:30.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:30.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:30.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:30.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:30.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:30.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:30.493+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:30.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:30.493+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:30.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:30.493+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:30.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:30.493+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:30.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:30.709+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:30.710519+0000) 2024-02-20T09:19:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:30.962+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:30.963623+0000) 2024-02-20T09:19:31.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:31.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:31.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:31.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:31.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:31.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:31.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:31.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:31.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:31.223+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.224396+0000) 2024-02-20T09:19:31.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:31.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:31.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:31.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:31.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:31.271+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:31.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:31.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:31.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:31.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:31.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:31.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:31.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:31.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:31.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:31.532+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:31.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:31.710+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.711891+0000) 2024-02-20T09:19:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:31.964+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.965126+0000) 2024-02-20T09:19:32.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:32.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.226143+0000) 2024-02-20T09:19:32.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:32.253+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.254809+0000) 2024-02-20T09:19:32.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:32.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:32.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:32.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:32.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:32.313+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:32.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:32.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:32.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:32.252+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.253845+0000) 2024-02-20T09:19:32.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:32.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:32.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:32.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:32.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:32.335+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:32.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:32.336+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:32.498 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:32.257+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61656 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:32.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:32.252+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.253851+0000) 2024-02-20T09:19:32.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:32.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:32.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:32.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:32.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:32.494+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:32.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:32.495+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:32.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:32.711+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.712096+0000) 2024-02-20T09:19:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:32.964+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.965299+0000) 2024-02-20T09:19:33.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:33.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:33.226335+0000) 2024-02-20T09:19:33.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:33.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:33.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:33.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:33.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:33.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:33.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:33.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:33.489 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:33.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:33.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:33.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:33.486+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:33.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:33.825 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:33.714+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:33.715509+0000) 2024-02-20T09:19:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:33.966+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:33.967837+0000) 2024-02-20T09:19:34.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:34.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:34.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:34.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:34.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:34.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:34.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:34.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:34.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:34.227+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:34.228701+0000) 2024-02-20T09:19:34.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:34.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:34.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:34.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:34.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:34.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:34.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:34.359+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:34.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:34.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:34.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:34.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:34.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:34.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:34.593 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:34.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:34.717+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:34.718227+0000) 2024-02-20T09:19:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:34.969+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:34.970244+0000) 2024-02-20T09:19:35.626 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:35.230+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:35.231394+0000) 2024-02-20T09:19:35.626 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:35.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:35.626 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:35.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:35.626 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:35.404+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:35.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:35.405+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:35.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:35.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:35.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:35.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:35.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:35.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:35.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:35.395+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:35.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:35.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:35.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:35.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:35.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:35.425+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:35.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:35.426+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:35.719+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:35.720740+0000) 2024-02-20T09:19:36.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:35.972+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:35.973088+0000) 2024-02-20T09:19:36.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:36.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:36.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:36.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:36.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:36.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:36.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:36.354+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:36.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:36.233+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.234853+0000) 2024-02-20T09:19:36.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:36.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:36.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:36.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:36.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:36.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:36.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:36.365+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:36.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:36.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:36.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:36.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:36.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:36.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:36.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:36.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:36.721+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.722471+0000) 2024-02-20T09:19:37.237 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:36.974+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.975267+0000) 2024-02-20T09:19:37.513 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:37.264+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61695 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:37.260+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.261759+0000) 2024-02-20T09:19:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:37.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:37.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:37.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:37.351+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:37.260+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.261654+0000) 2024-02-20T09:19:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:37.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:37.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:37.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:37.418+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:37.235+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.236446+0000) 2024-02-20T09:19:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:37.262+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.262943+0000) 2024-02-20T09:19:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:37.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:37.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:37.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:37.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:37.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:37.721+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.722663+0000) 2024-02-20T09:19:38.237 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:37.974+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.975487+0000) 2024-02-20T09:19:38.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:38.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:38.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:38.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:38.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:38.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:38.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:38.451+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:38.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:38.235+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:38.236645+0000) 2024-02-20T09:19:38.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:38.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:38.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:38.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:38.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:38.353+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:38.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:38.353+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:38.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:38.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:38.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:38.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:38.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:38.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:38.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:38.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:38.724+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:38.725107+0000) 2024-02-20T09:19:39.240 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:38.976+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:38.977732+0000) 2024-02-20T09:19:39.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:39.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:39.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:39.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:39.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:39.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:39.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:39.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:39.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:39.238+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:39.239896+0000) 2024-02-20T09:19:39.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:39.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:39.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:39.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:39.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:39.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:39.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:39.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:39.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:39.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:39.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:39.500+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:39.726+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:39.727531+0000) 2024-02-20T09:19:40.243 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:39.980+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:39.981067+0000) 2024-02-20T09:19:40.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:40.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:40.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:40.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:40.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:40.349+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:40.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:40.350+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:40.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:40.241+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:40.242124+0000) 2024-02-20T09:19:40.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:40.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:40.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:40.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:40.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:40.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:40.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:40.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:40.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:40.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:40.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:40.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:40.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:40.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:40.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:40.452+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:40.729+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:40.730119+0000) 2024-02-20T09:19:41.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:40.983+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:40.983969+0000) 2024-02-20T09:19:41.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:41.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:41.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:41.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:41.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:41.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:41.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:41.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:41.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:41.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:41.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:41.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:41.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:41.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:41.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:41.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:41.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:41.243+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.244688+0000) 2024-02-20T09:19:41.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:41.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:41.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:41.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:41.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:41.310+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:41.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:41.311+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:41.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:41.730+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.731591+0000) 2024-02-20T09:19:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:41.984+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.985747+0000) 2024-02-20T09:19:42.626 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:42.272+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61735 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:42.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:42.269+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.270742+0000) 2024-02-20T09:19:42.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:42.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:42.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:42.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:42.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:42.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:42.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:42.387+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:42.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:42.268+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.269625+0000) 2024-02-20T09:19:42.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:42.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:42.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:42.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:42.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:42.417+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:42.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:42.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:42.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:42.246+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.246952+0000) 2024-02-20T09:19:42.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:42.269+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.270167+0000) 2024-02-20T09:19:42.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:42.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:42.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:42.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:42.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:42.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:42.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:42.285+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:42.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:42.730+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.731791+0000) 2024-02-20T09:19:43.248 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:42.985+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.985947+0000) 2024-02-20T09:19:43.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:43.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:43.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:43.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:43.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:43.434+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:43.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:43.435+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:43.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:43.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:43.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:43.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:43.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:43.431+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:43.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:43.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:43.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:43.246+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:43.247140+0000) 2024-02-20T09:19:43.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:43.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:43.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:43.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:43.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:43.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:43.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:43.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:43.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:43.733+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:43.734312+0000) 2024-02-20T09:19:44.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:43.987+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:43.988411+0000) 2024-02-20T09:19:44.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:44.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:44.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:44.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:44.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:44.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:44.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:44.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:44.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:44.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:44.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:44.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:44.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:44.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:44.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:44.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:44.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:44.249+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:44.250442+0000) 2024-02-20T09:19:44.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:44.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:44.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:44.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:44.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:44.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:44.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:44.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:44.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:44.735+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:44.736606+0000) 2024-02-20T09:19:45.254 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:44.990+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:44.991910+0000) 2024-02-20T09:19:45.254 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:45.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:45.254 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:45.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:45.254 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:45.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:45.254 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:45.239+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:45.627 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:45.252+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:45.253061+0000) 2024-02-20T09:19:45.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:45.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:45.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:45.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:45.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:45.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:45.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:45.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:45.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:45.397+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:45.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:45.397+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:45.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:45.397+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:45.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:45.398+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:45.739+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:45.740083+0000) 2024-02-20T09:19:46.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:45.994+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:45.995560+0000) 2024-02-20T09:19:46.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:46.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:46.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:46.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:46.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:46.250+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:46.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:46.251+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:46.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:46.255+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.256669+0000) 2024-02-20T09:19:46.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:46.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:46.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:46.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:46.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:46.411+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:46.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:46.411+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:46.879 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:46.741+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.742228+0000) 2024-02-20T09:19:46.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:46.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:46.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:46.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:46.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:46.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:46.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:46.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:46.996+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.997098+0000) 2024-02-20T09:19:47.509 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:47.280+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61774 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:47.257+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.258259+0000) 2024-02-20T09:19:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:47.277+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.278583+0000) 2024-02-20T09:19:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:47.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:47.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:47.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:47.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:47.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:47.277+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.278149+0000) 2024-02-20T09:19:47.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:47.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:47.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:47.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:47.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:47.395+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:47.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:47.396+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:47.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:47.276+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.277522+0000) 2024-02-20T09:19:47.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:47.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:47.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:47.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:47.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:47.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:47.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:47.741+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.742425+0000) 2024-02-20T09:19:48.259 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:47.996+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.997290+0000) 2024-02-20T09:19:48.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:48.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:48.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:48.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:48.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:48.493+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:48.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:48.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:48.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:48.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:48.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:48.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:48.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:48.445+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:48.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:48.446+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:48.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:48.257+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:48.258369+0000) 2024-02-20T09:19:48.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:48.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:48.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:48.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:48.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:48.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:48.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:48.284+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:48.743+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:48.744464+0000) 2024-02-20T09:19:49.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:48.998+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:48.999893+0000) 2024-02-20T09:19:49.627 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:49.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:49.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:49.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:49.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:49.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:49.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:49.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:49.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:49.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:49.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:49.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:49.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:49.442+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:49.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:49.443+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:49.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:49.259+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:49.260604+0000) 2024-02-20T09:19:49.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:49.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:49.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:49.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:49.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:49.331+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:49.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:49.332+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:49.746+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:49.747837+0000) 2024-02-20T09:19:50.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:50.002+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:50.003517+0000) 2024-02-20T09:19:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:50.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:50.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:50.473+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:50.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:50.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:50.261+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:50.262904+0000) 2024-02-20T09:19:50.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:50.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:50.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:50.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:50.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:50.370+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:50.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:50.371+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:50.750+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:50.751505+0000) 2024-02-20T09:19:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:50.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:50.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:50.515+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:50.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:51.266 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:51.004+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.005646+0000) 2024-02-20T09:19:51.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:51.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:51.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:51.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:51.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:51.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:51.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:51.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:51.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:51.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:51.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:51.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:51.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:51.488+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:51.629 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:51.488+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:51.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:51.264+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.265541+0000) 2024-02-20T09:19:51.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:51.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:51.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:51.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:51.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:51.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:51.752+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.753873+0000) 2024-02-20T09:19:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:52.006+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.007048+0000) 2024-02-20T09:19:52.523 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:52.287+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61814 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:52.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:52.284+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.285507+0000) 2024-02-20T09:19:52.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:52.287+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.288849+0000) 2024-02-20T09:19:52.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:52.266+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.267722+0000) 2024-02-20T09:19:52.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:52.284+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.285912+0000) 2024-02-20T09:19:52.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:52.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:52.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:52.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:52.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:52.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:52.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:52.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:52.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:52.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:52.525+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:52.525+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:52.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:52.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:52.521+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:52.522+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:52.753+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.754074+0000) 2024-02-20T09:19:53.268 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:53.006+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:53.007239+0000) 2024-02-20T09:19:53.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:53.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:53.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:53.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:53.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:53.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:53.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:53.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:53.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:53.266+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:53.267922+0000) 2024-02-20T09:19:53.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:53.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:53.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:53.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:53.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:53.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:53.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:53.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:53.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:53.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:53.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:53.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:53.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:53.524+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:53.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:53.525+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:53.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:53.756+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:53.757556+0000) 2024-02-20T09:19:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:54.008+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:54.009472+0000) 2024-02-20T09:19:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:54.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:54.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:54.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:54.466+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:54.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:54.269+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:54.270314+0000) 2024-02-20T09:19:54.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:54.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:54.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:54.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:54.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:54.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:54.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:54.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:54.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:54.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:54.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:54.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:54.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:54.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:54.539+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:54.758+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:54.759825+0000) 2024-02-20T09:19:55.273 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:55.011+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:55.012782+0000) 2024-02-20T09:19:55.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:55.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:55.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:55.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:55.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:55.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:55.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:55.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:55.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:55.271+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:55.272465+0000) 2024-02-20T09:19:55.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:55.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:55.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:55.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:55.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:55.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:55.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:55.498+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:55.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:55.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:55.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:55.586+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:55.762+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:55.763166+0000) 2024-02-20T09:19:56.276 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:56.015+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.016200+0000) 2024-02-20T09:19:56.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:56.274+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.275241+0000) 2024-02-20T09:19:56.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:56.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:56.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:56.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:56.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:56.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:56.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:56.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:56.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:56.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:56.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:56.545+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:56.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:56.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:56.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:56.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:56.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:56.763+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.764863+0000) 2024-02-20T09:19:57.277 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:57.017+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.017955+0000) 2024-02-20T09:19:57.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:19:57.295+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61853 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:19:57.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:57.292+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.293568+0000) 2024-02-20T09:19:57.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:57.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:57.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:57.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:57.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:57.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:57.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:57.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:57.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:57.275+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.276792+0000) 2024-02-20T09:19:57.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:57.291+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.292666+0000) 2024-02-20T09:19:57.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:57.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:57.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:57.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:57.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:57.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:57.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:57.512+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:57.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:57.292+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.293059+0000) 2024-02-20T09:19:57.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:57.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:57.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:57.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:57.582 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:57.577+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:57.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:57.764+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.765057+0000) 2024-02-20T09:19:58.277 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:58.017+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:58.018143+0000) 2024-02-20T09:19:58.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:58.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:58.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:58.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:58.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:58.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:58.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:58.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:58.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:58.276+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:58.277093+0000) 2024-02-20T09:19:58.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:58.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:58.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:58.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:58.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:58.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:58.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:58.500+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:58.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:58.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:58.561+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:58.562+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:58.766+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:58.767230+0000) 2024-02-20T09:19:59.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:59.019+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:59.020117+0000) 2024-02-20T09:19:59.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:59.498+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:59.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:59.498+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:59.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:59.498+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:19:59.561 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:19:59.499+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:59.561 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:59.278+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:59.279184+0000) 2024-02-20T09:19:59.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:59.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:59.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:59.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:59.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:59.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:19:59.562 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:19:59.461+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:59.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:59.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:59.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:19:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:59.560+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:19:59.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:19:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:19:59.768+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:59.769313+0000) 2024-02-20T09:20:00.283 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:00.022+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:00.023579+0000) 2024-02-20T09:20:00.534 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:00.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:00.534 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:00.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:00.534 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:00.531+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:00.534 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:00.532+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:00.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:00.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:00.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:00.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:00.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:00.492+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:00.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:00.492+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:00.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:00.281+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:00.282621+0000) 2024-02-20T09:20:00.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:00.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:00.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:00.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:00.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:00.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:00.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:00.416+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:00.770+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:00.771477+0000) 2024-02-20T09:20:01.287 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:01.026+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.027138+0000) 2024-02-20T09:20:01.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:01.285+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.286286+0000) 2024-02-20T09:20:01.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:01.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:01.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:01.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:01.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:01.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:01.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:01.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:01.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:01.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:01.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:01.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:01.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:01.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:01.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:01.501+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:01.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:01.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:01.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:01.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:01.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:01.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:01.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:01.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:01.947 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:01.772+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.772974+0000) 2024-02-20T09:20:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:02.028+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.029470+0000) 2024-02-20T09:20:02.496 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:02.302+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61893 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:02.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:02.299+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.300456+0000) 2024-02-20T09:20:02.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:02.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:02.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:02.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:02.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:02.495+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:02.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:02.286+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.287780+0000) 2024-02-20T09:20:02.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:02.300+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.301731+0000) 2024-02-20T09:20:02.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:02.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:02.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:02.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:02.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:02.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:02.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:02.421+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:02.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:02.299+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.300716+0000) 2024-02-20T09:20:02.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:02.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:02.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:02.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:02.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:02.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:02.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:02.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:02.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:02.496+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:03.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:02.772+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.773167+0000) 2024-02-20T09:20:03.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:03.028+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:03.029646+0000) 2024-02-20T09:20:03.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:03.287+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:03.288017+0000) 2024-02-20T09:20:03.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:03.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:03.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:03.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:03.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:03.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:03.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:03.431+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:03.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:03.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:03.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:03.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:03.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:03.480+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:03.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:03.481+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:03.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:03.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:03.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:03.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:03.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:03.474+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:03.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:03.474+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:03.774+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:03.775216+0000) 2024-02-20T09:20:04.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:04.031+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:04.032121+0000) 2024-02-20T09:20:04.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:04.290+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:04.291337+0000) 2024-02-20T09:20:04.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:04.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:04.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:04.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:04.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:04.467+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:04.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:04.468+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:04.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:04.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:04.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:04.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:04.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:04.488+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:04.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:04.489+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:04.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:04.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:04.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:04.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:04.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:04.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:04.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:04.523+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:04.776+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:04.777673+0000) 2024-02-20T09:20:05.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:05.033+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:05.034807+0000) 2024-02-20T09:20:05.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:05.293+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:05.294667+0000) 2024-02-20T09:20:05.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:05.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:05.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:05.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:05.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:05.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:05.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:05.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:05.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:05.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:05.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:05.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:05.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:05.530+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:05.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:05.530+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:05.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:05.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:05.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:05.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:05.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:05.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:05.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:05.540+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:05.780+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:05.781016+0000) 2024-02-20T09:20:06.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:06.036+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.037299+0000) 2024-02-20T09:20:06.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:06.295+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.297264+0000) 2024-02-20T09:20:06.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:06.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:06.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:06.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:06.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:06.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:06.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:06.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:06.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:06.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:06.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:06.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:06.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:06.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:06.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:06.519+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:06.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:06.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:06.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:06.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:06.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:06.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:06.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:06.531+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:06.780+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.782584+0000) 2024-02-20T09:20:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:07.037+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.039092+0000) 2024-02-20T09:20:07.629 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:07.309+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61932 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:07.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:07.308+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.310127+0000) 2024-02-20T09:20:07.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:07.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:07.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:07.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:07.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:07.495+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:07.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:07.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:07.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:07.306+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.308095+0000) 2024-02-20T09:20:07.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:07.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:07.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:07.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:07.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:07.530+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:07.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:07.531+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:07.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:07.297+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.298903+0000) 2024-02-20T09:20:07.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:07.306+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.308744+0000) 2024-02-20T09:20:07.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:07.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:07.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:07.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:07.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:07.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:07.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:07.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:07.781+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.782821+0000) 2024-02-20T09:20:08.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:08.038+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:08.039261+0000) 2024-02-20T09:20:08.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:08.298+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:08.299165+0000) 2024-02-20T09:20:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:08.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:08.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:08.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:08.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:08.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:08.555 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:08.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:08.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:08.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:08.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:08.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:08.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:08.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:08.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:08.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:08.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:08.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:08.552+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:08.783+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:08.785204+0000) 2024-02-20T09:20:09.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:09.039+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:09.041918+0000) 2024-02-20T09:20:09.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:09.301+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:09.302027+0000) 2024-02-20T09:20:09.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:09.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:09.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:09.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:09.566 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:09.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:09.567 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:09.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:09.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:09.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:09.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:09.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:09.786+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:09.787733+0000) 2024-02-20T09:20:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:09.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:09.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:09.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:09.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:09.565+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:10.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:10.044+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:10.045093+0000) 2024-02-20T09:20:10.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:10.303+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:10.304588+0000) 2024-02-20T09:20:10.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:10.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:10.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:10.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:10.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:10.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:10.569 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:10.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:10.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:10.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:10.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:10.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:10.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:10.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:10.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:10.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:10.789+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:10.790258+0000) 2024-02-20T09:20:11.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:11.046+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.047646+0000) 2024-02-20T09:20:11.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:11.306+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.307011+0000) 2024-02-20T09:20:11.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:11.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:11.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:11.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:11.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:11.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:11.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:11.451+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:11.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:11.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:11.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:11.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:11.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:11.575+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:11.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:11.576+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:11.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:11.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:11.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:11.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:11.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:11.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:11.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:11.595+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:11.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:11.791+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.791958+0000) 2024-02-20T09:20:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:12.049+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.050028+0000) 2024-02-20T09:20:12.574 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:12.308+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.308922+0000) 2024-02-20T09:20:12.574 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:12.314+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.315567+0000) 2024-02-20T09:20:12.574 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:12.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:12.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:12.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:12.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:12.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:12.575 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:12.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:12.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:12.315+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.316569+0000) 2024-02-20T09:20:12.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:12.314+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.315620+0000) 2024-02-20T09:20:12.576 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:12.318+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 61972 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:12.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:12.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:12.624+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:12.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:12.791+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.792126+0000) 2024-02-20T09:20:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:12.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:12.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:12.572+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:12.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:12.573+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:13.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:13.049+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:13.050229+0000) 2024-02-20T09:20:13.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:13.308+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:13.309166+0000) 2024-02-20T09:20:13.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:13.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:13.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:13.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:13.590 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:13.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:13.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:13.486+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:13.665+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:13.665+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:13.665+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:13.666+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:13.794+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:13.795018+0000) 2024-02-20T09:20:13.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:13.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:13.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:13.587+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:13.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:13.588+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:14.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:14.051+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:14.052929+0000) 2024-02-20T09:20:14.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:14.310+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:14.311724+0000) 2024-02-20T09:20:14.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:14.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:14.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:14.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:14.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:14.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:14.603 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:14.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:14.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:14.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:14.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:14.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:14.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:14.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:14.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:14.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:14.673+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:14.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:14.797+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:14.798259+0000) 2024-02-20T09:20:15.307 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:15.054+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:15.055387+0000) 2024-02-20T09:20:15.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:15.313+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:15.314203+0000) 2024-02-20T09:20:15.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:15.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:15.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:15.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:15.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:15.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:15.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:15.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:15.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:15.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:15.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:15.641+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:15.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:15.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:15.626+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:15.627+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:15.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:15.799+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:15.800738+0000) 2024-02-20T09:20:16.317 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:16.057+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.057949+0000) 2024-02-20T09:20:16.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:16.315+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.316554+0000) 2024-02-20T09:20:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:16.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:16.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:16.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:16.575+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:16.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:16.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:16.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:16.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:16.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:16.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:16.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:16.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:16.611+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:16.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:16.801+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.802709+0000) 2024-02-20T09:20:17.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:17.058+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.059545+0000) 2024-02-20T09:20:17.574 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:17.326+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62011 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:17.575 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:17.323+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.323995+0000) 2024-02-20T09:20:17.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:17.323+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.324209+0000) 2024-02-20T09:20:17.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:17.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:17.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:17.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:17.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:17.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:17.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:17.318+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.319178+0000) 2024-02-20T09:20:17.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:17.323+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.324787+0000) 2024-02-20T09:20:17.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:17.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:17.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:17.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:17.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:17.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:17.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:17.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:17.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:17.574+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:17.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:17.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:17.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:17.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:17.802+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.802977+0000) 2024-02-20T09:20:18.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:18.058+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:18.059735+0000) 2024-02-20T09:20:18.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:18.318+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:18.319397+0000) 2024-02-20T09:20:18.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:18.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:18.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:18.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:18.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:18.502+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:18.573 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:18.503+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:18.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:18.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:18.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:18.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:18.571+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:18.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:18.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:18.599+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:18.599+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:18.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:18.804+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:18.805503+0000) 2024-02-20T09:20:19.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:19.060+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:19.061847+0000) 2024-02-20T09:20:19.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:19.321+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:19.322106+0000) 2024-02-20T09:20:19.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:19.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:19.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:19.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:19.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:19.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:19.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:19.507+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:19.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:19.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:19.599+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:19.600+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:19.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:19.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:19.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:19.586+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:19.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:19.807+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:19.808101+0000) 2024-02-20T09:20:20.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:20.063+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:20.063931+0000) 2024-02-20T09:20:20.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:20.323+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:20.324451+0000) 2024-02-20T09:20:20.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:20.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:20.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:20.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:20.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:20.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:20.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:20.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:20.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:20.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:20.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:20.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:20.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:20.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:20.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:20.566+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:20.632 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:20.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:20.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:20.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:20.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:20.569+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:20.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:20.570+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:20.809+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:20.810179+0000) 2024-02-20T09:20:21.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:21.066+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.067524+0000) 2024-02-20T09:20:21.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:21.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:21.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:21.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:21.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:21.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:21.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:21.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:21.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:21.325+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.326750+0000) 2024-02-20T09:20:21.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:21.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:21.595 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:21.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:21.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:21.460+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:21.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:21.461+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:21.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:21.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:21.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:21.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:21.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:21.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:21.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:21.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:21.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:21.811+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.812497+0000) 2024-02-20T09:20:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:22.068+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.069077+0000) 2024-02-20T09:20:22.631 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:22.334+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62051 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:22.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:22.332+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.333911+0000) 2024-02-20T09:20:22.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:22.332+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.333370+0000) 2024-02-20T09:20:22.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:22.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:22.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:22.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:22.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:22.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:22.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:22.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:22.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:22.330+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.331743+0000) 2024-02-20T09:20:22.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:22.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:22.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:22.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:22.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:22.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:22.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:22.495+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:22.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:22.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:22.640+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:22.641+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:22.811+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.812679+0000) 2024-02-20T09:20:23.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:23.068+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:23.069277+0000) 2024-02-20T09:20:23.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:23.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:23.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:23.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:23.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:23.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:23.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:23.421+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:23.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:23.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:23.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:23.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:23.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:23.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:23.710 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:23.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:23.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:23.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:23.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:23.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:23.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:23.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:23.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:23.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:23.813+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:23.814784+0000) 2024-02-20T09:20:24.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:24.071+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:24.072205+0000) 2024-02-20T09:20:24.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:24.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:24.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:24.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:24.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:24.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:24.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:24.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:24.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:24.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:24.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:24.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:24.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:24.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:24.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:24.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:24.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:24.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:24.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:24.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:24.815+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:24.816897+0000) 2024-02-20T09:20:25.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:25.073+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:25.074463+0000) 2024-02-20T09:20:25.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:25.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:25.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:25.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:25.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:25.343+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:25.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:25.344+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:25.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:25.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:25.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:25.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:25.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:25.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:25.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:25.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:25.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:25.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:25.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:25.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:25.818+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:25.819510+0000) 2024-02-20T09:20:26.331 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:26.075+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.076642+0000) 2024-02-20T09:20:26.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:26.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:26.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:26.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:26.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:26.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:26.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:26.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:26.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:26.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:26.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:26.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:26.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:26.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:26.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:26.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:26.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:26.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:26.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:26.655+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:26.820+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.820970+0000) 2024-02-20T09:20:27.337 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:27.078+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.078999+0000) 2024-02-20T09:20:27.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:27.341+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62090 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:27.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:27.338+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.339091+0000) 2024-02-20T09:20:27.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:27.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:27.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:27.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:27.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:27.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:27.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:27.349+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:27.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:27.339+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.339980+0000) 2024-02-20T09:20:27.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:27.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:27.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:27.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:27.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:27.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:27.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:27.509+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:27.611 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:27.340+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.341488+0000) 2024-02-20T09:20:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:27.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:27.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:27.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:27.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:27.820+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.821151+0000) 2024-02-20T09:20:28.339 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:28.078+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:28.079188+0000) 2024-02-20T09:20:28.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:28.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:28.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:28.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:28.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:28.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:28.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:28.339+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:28.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:28.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:28.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:28.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:28.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:28.539+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:28.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:28.540+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:28.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:28.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:28.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:28.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:28.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:28.822+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:28.823335+0000) 2024-02-20T09:20:29.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:29.080+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:29.081802+0000) 2024-02-20T09:20:29.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:29.077+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:29.078677+0000) 2024-02-20T09:20:29.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:29.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:29.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:29.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:29.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:29.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:29.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:29.315+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:29.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:29.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:29.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:29.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:29.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:29.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:29.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:29.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:29.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:29.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:29.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:29.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:29.825+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:29.826514+0000) 2024-02-20T09:20:30.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:30.084+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:30.085420+0000) 2024-02-20T09:20:30.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:30.081+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:30.082225+0000) 2024-02-20T09:20:30.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:30.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:30.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:30.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:30.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:30.277+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:30.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:30.278+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:30.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:30.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:30.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:30.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:30.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:30.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:30.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:30.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:30.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:30.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:30.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:30.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:30.827+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:30.828911+0000) 2024-02-20T09:20:31.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:31.087+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.088221+0000) 2024-02-20T09:20:31.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:31.084+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.085572+0000) 2024-02-20T09:20:31.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:31.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:31.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:31.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:31.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:31.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:31.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:31.236+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:31.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:31.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:31.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:31.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:31.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:31.581+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:31.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:31.581+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:31.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:31.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:31.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:31.709+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:31.830+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.831228+0000) 2024-02-20T09:20:32.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:32.090+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.091406+0000) 2024-02-20T09:20:32.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:32.088+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.089133+0000) 2024-02-20T09:20:32.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:32.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:32.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:32.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:32.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:32.206+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:32.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:32.207+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:32.632 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:32.350+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62130 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:32.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:32.345+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.346668+0000) 2024-02-20T09:20:32.633 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:32.345+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.346871+0000) 2024-02-20T09:20:32.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:32.347+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.348817+0000) 2024-02-20T09:20:32.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:32.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:32.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:32.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:32.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:32.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:32.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:32.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:32.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:32.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:32.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:32.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:32.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:32.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:32.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:32.709+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:32.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:32.830+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.831425+0000) 2024-02-20T09:20:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:33.090+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:33.091610+0000) 2024-02-20T09:20:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:33.089+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:33.090627+0000) 2024-02-20T09:20:33.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:33.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:33.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:33.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:33.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:33.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:33.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:33.247+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:33.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:33.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:33.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:33.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:33.828 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:33.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:33.829 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:33.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:33.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:33.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:33.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:33.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:33.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:33.734+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:33.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:33.735+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:34.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:33.833+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:33.834719+0000) 2024-02-20T09:20:34.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:34.094+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:34.095097+0000) 2024-02-20T09:20:34.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:34.092+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:34.093090+0000) 2024-02-20T09:20:34.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:34.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:34.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:34.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:34.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:34.199+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:34.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:34.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:34.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:34.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:34.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:34.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:34.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:34.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:34.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:34.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:34.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:34.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:34.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:34.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:34.835+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:34.836818+0000) 2024-02-20T09:20:35.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:35.097+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:35.098636+0000) 2024-02-20T09:20:35.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:35.094+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:35.095286+0000) 2024-02-20T09:20:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:35.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:35.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:35.174+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:35.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:35.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:35.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:35.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:35.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:35.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:35.580+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:35.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:35.581+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:35.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:35.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:35.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:35.740+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:35.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:35.741+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:35.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:35.838+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:35.839586+0000) 2024-02-20T09:20:36.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:36.101+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.102218+0000) 2024-02-20T09:20:36.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:36.097+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.098353+0000) 2024-02-20T09:20:36.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:36.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:36.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:36.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:36.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:36.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:36.311 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:36.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:36.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:36.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:36.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:36.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:36.765+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:36.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:36.841+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.842155+0000) 2024-02-20T09:20:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:36.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:36.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:36.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:36.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:36.620+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:37.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:37.102+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.103618+0000) 2024-02-20T09:20:37.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:37.099+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.100200+0000) 2024-02-20T09:20:37.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:37.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:37.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:37.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:37.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:37.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:37.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:37.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:37.632 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:37.358+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62169 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:37.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:37.356+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.357285+0000) 2024-02-20T09:20:37.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:37.354+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.355495+0000) 2024-02-20T09:20:37.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:37.355+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.356111+0000) 2024-02-20T09:20:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:37.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:37.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:37.636+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:37.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:37.636+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:37.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:37.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:37.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:37.732+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:37.841+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.842350+0000) 2024-02-20T09:20:38.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:38.102+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:38.103807+0000) 2024-02-20T09:20:38.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:38.099+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:38.100408+0000) 2024-02-20T09:20:38.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:38.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:38.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:38.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:38.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:38.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:38.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:38.208+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:38.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:38.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:38.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:38.652+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:38.718+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:38.718+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:38.718+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:38.719+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:38.843+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:38.844326+0000) 2024-02-20T09:20:39.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:39.106+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:39.107079+0000) 2024-02-20T09:20:39.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:39.103+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:39.103933+0000) 2024-02-20T09:20:39.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:39.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:39.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:39.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:39.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:39.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:39.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:39.163+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:39.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:39.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:39.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:39.732+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:39.845+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:39.846730+0000) 2024-02-20T09:20:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:39.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:39.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:39.691+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:39.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:39.692+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:40.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:40.109+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:40.110413+0000) 2024-02-20T09:20:40.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:40.106+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:40.107435+0000) 2024-02-20T09:20:40.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:40.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:40.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:40.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:40.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:40.143+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:40.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:40.144+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:40.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:40.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:40.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:40.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:40.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:40.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:40.715+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:40.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:40.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:40.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:40.726+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:40.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:40.847+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:40.848921+0000) 2024-02-20T09:20:41.384 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:41.112+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.113909+0000) 2024-02-20T09:20:41.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:41.110+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.111020+0000) 2024-02-20T09:20:41.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:41.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:41.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:41.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:41.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:41.167+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:41.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:41.167+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:41.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:41.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:41.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:41.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:41.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:41.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:41.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:41.688+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:41.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:41.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:41.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:41.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:41.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:41.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:41.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:41.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:41.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:41.850+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.851315+0000) 2024-02-20T09:20:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:42.114+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.115497+0000) 2024-02-20T09:20:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:42.111+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.112601+0000) 2024-02-20T09:20:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:42.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:42.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:42.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:42.141+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:42.633 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:42.366+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62209 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:42.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:42.362+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.363206+0000) 2024-02-20T09:20:42.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:42.364+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.365112+0000) 2024-02-20T09:20:42.634 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:42.361+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.362721+0000) 2024-02-20T09:20:42.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:42.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:42.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:42.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:42.704+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:42.850+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.851513+0000) 2024-02-20T09:20:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:42.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:42.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:42.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:42.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:42.703+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:43.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:43.114+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:43.115678+0000) 2024-02-20T09:20:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:43.111+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:43.112798+0000) 2024-02-20T09:20:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:43.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:43.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:43.162+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:43.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:43.163+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:43.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:43.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:43.702+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:43.703+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:43.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:43.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:43.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:43.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:43.681+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:43.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:43.852+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:43.853610+0000) 2024-02-20T09:20:44.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:44.118+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:44.119153+0000) 2024-02-20T09:20:44.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:44.114+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:44.115312+0000) 2024-02-20T09:20:44.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:44.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:44.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:44.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:44.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:45.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:44.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:45.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:44.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:45.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:44.752+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:45.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:44.753+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:45.120 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:44.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:45.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:44.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:45.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:44.725+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:45.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:44.725+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:45.121 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:44.855+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:44.856145+0000) 2024-02-20T09:20:45.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:45.121+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:45.122729+0000) 2024-02-20T09:20:45.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:45.117+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:45.118884+0000) 2024-02-20T09:20:45.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:45.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:45.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:45.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:45.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:45.220+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:45.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:45.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:46.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:45.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:46.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:45.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:46.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:45.741+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:46.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:45.741+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:46.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:45.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:46.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:45.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:46.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:45.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:46.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:45.733+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:46.124 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:45.858+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:45.859741+0000) 2024-02-20T09:20:46.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:46.123+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.124886+0000) 2024-02-20T09:20:46.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:46.121+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.122306+0000) 2024-02-20T09:20:46.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:46.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:46.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:46.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:46.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:46.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:46.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:46.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:47.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:46.744+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:47.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:46.744+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:47.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:46.744+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:47.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:46.745+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:47.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:46.860+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.861456+0000) 2024-02-20T09:20:47.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:46.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:47.126 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:46.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:47.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:46.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:47.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:46.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:47.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:47.370+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.371544+0000) 2024-02-20T09:20:47.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:47.125+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.126801+0000) 2024-02-20T09:20:47.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:47.371+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.371975+0000) 2024-02-20T09:20:47.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:47.123+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.124518+0000) 2024-02-20T09:20:47.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:47.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:47.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:47.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:47.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:47.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:47.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:47.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:47.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:47.371+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.372525+0000) 2024-02-20T09:20:47.633 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:47.373+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62248 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:47.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:47.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:47.764+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:47.765+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:47.860+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.861673+0000) 2024-02-20T09:20:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:47.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:47.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:47.768+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:47.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:47.769+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:48.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:48.126+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:48.126957+0000) 2024-02-20T09:20:48.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:48.123+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:48.124726+0000) 2024-02-20T09:20:48.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:48.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:48.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:48.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:48.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:48.227+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:48.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:48.227+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:49.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:48.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:49.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:48.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:49.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:48.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:49.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:48.740+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:49.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:48.864+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:48.865265+0000) 2024-02-20T09:20:49.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:48.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:49.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:48.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:49.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:48.775+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:49.129 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:48.776+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:49.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:49.128+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:49.129549+0000) 2024-02-20T09:20:49.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:49.126+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:49.127215+0000) 2024-02-20T09:20:49.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:49.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:49.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:49.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:49.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:49.252+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:49.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:49.253+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:49.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:49.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:49.772+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:49.773+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:49.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:49.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:49.695+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:49.696+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:49.866+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:49.867443+0000) 2024-02-20T09:20:50.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:50.131+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:50.132094+0000) 2024-02-20T09:20:50.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:50.128+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:50.129792+0000) 2024-02-20T09:20:50.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:50.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:50.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:50.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:50.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:50.218+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:50.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:50.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:50.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:50.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:50.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:50.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:50.868+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:50.869549+0000) 2024-02-20T09:20:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:50.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:50.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:50.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:50.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:50.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:51.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:51.134+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.135691+0000) 2024-02-20T09:20:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:51.132+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.133395+0000) 2024-02-20T09:20:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:51.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:51.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:51.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:51.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:51.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:51.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:51.761+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:51.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:51.761+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:51.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:51.761+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:51.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:51.762+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:51.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:51.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:51.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:51.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:51.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:51.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:51.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:51.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:51.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:51.870+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.871207+0000) 2024-02-20T09:20:52.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:52.133+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.134912+0000) 2024-02-20T09:20:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:52.136+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.137367+0000) 2024-02-20T09:20:52.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:52.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:52.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:52.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:52.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:52.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:52.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:52.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:52.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:52.378+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.379439+0000) 2024-02-20T09:20:52.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:52.377+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.378763+0000) 2024-02-20T09:20:52.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:52.377+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.378894+0000) 2024-02-20T09:20:52.503 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:52.381+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62288 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:52.757 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:52.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:52.757 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:52.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:52.757 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:52.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:52.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:52.679+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:53.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:52.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:53.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:52.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:53.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:52.755+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:53.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:52.756+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:53.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:52.870+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.871389+0000) 2024-02-20T09:20:53.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:53.136+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:53.137547+0000) 2024-02-20T09:20:53.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:53.134+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:53.135101+0000) 2024-02-20T09:20:53.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:53.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:53.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:53.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:53.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:53.195+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:53.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:53.196+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:53.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:53.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:53.792+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:53.793+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:53.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:53.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:53.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:53.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:53.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:53.873+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:53.874635+0000) 2024-02-20T09:20:54.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:54.140+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:54.141294+0000) 2024-02-20T09:20:54.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:54.137+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:54.138375+0000) 2024-02-20T09:20:54.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:54.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:54.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:54.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:54.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:54.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:54.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:54.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:54.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:54.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:54.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:54.659+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:55.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:54.877+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:54.878060+0000) 2024-02-20T09:20:55.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:54.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:55.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:54.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:55.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:54.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:55.143 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:54.835+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:55.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:55.142+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:55.143837+0000) 2024-02-20T09:20:55.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:55.140+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:55.141118+0000) 2024-02-20T09:20:55.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:55.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:55.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:55.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:55.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:55.212+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:55.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:55.213+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:55.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:55.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:55.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:55.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:55.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:55.681+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:55.681+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:55.681+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:55.682+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:55.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:55.880+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:55.881427+0000) 2024-02-20T09:20:56.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:56.145+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.146096+0000) 2024-02-20T09:20:56.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:56.143+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.144368+0000) 2024-02-20T09:20:56.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:56.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:56.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:56.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:56.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:56.202+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:56.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:56.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:56.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:56.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:56.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:56.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:57.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:56.882+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.883902+0000) 2024-02-20T09:20:57.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:56.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:57.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:56.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:57.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:56.910+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:57.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:56.911+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:57.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:57.145+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.146756+0000) 2024-02-20T09:20:57.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:57.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:57.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:57.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:57.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:57.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:57.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:57.219+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:57.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:57.386+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.387782+0000) 2024-02-20T09:20:57.408 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:20:57.389+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62327 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:20:57.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:57.386+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.387914+0000) 2024-02-20T09:20:57.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:57.147+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.148101+0000) 2024-02-20T09:20:57.409 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:57.386+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.387082+0000) 2024-02-20T09:20:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:57.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:57.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:57.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:57.929+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:57.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:57.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:57.692+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:57.693+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:57.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:57.883+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.884112+0000) 2024-02-20T09:20:58.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:58.147+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:58.148300+0000) 2024-02-20T09:20:58.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:58.146+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:58.146955+0000) 2024-02-20T09:20:58.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:58.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:58.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:58.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:58.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:58.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:58.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:58.180+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:59.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:58.944+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:59.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:58.944+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:59.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:58.944+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:20:59.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:58.945+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:59.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:58.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:59.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:58.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:59.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:58.735+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:20:59.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:58.736+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:20:59.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:58.885+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:58.886848+0000) 2024-02-20T09:20:59.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:59.149+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:59.151819+0000) 2024-02-20T09:20:59.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:59.147+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:59.149521+0000) 2024-02-20T09:20:59.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:59.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:59.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:59.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:59.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:59.207+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:20:59.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:20:59.208+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:00.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:59.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:00.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:59.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:00.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:59.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:00.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:20:59.984+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:00.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:59.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:00.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:59.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:00.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:59.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:00.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:59.738+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:00.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:20:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:20:59.887+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:59.889445+0000) 2024-02-20T09:21:00.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:00.153+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:00.155427+0000) 2024-02-20T09:21:00.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:00.150+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:00.152912+0000) 2024-02-20T09:21:00.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:00.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:00.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:00.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:00.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:00.245+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:00.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:00.246+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:01.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:00.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:01.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:00.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:01.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:00.753+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:01.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:00.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:01.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:00.891+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:00.892104+0000) 2024-02-20T09:21:01.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:01.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:01.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:01.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:01.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:01.035+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:01.035+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:01.156+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.158099+0000) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:01.155+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.156032+0000) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:01.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:01.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:01.208+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:01.314 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:01.209+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:01.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:01.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:01.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:01.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:01.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:01.804+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:01.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:01.804+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:02.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:02.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:02.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:02.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:02.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:02.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:02.159 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:02.043+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:02.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:01.892+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.894691+0000) 2024-02-20T09:21:02.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:02.393+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.395486+0000) 2024-02-20T09:21:02.413 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:02.396+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62367 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:02.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:02.157+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.159928+0000) 2024-02-20T09:21:02.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:02.393+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.395567+0000) 2024-02-20T09:21:02.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:02.156+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.157681+0000) 2024-02-20T09:21:02.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:02.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:02.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:02.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:02.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:02.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:02.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:02.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:02.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:02.393+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.395237+0000) 2024-02-20T09:21:03.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:02.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:03.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:02.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:03.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:02.812+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:03.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:02.813+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:03.089 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:02.894+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.894931+0000) 2024-02-20T09:21:03.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:03.157+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:03.157949+0000) 2024-02-20T09:21:03.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:03.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:03.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:03.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:03.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:03.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:03.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:03.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:03.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:03.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:03.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:03.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:03.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:03.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:03.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:03.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:03.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:03.159+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:03.160116+0000) 2024-02-20T09:21:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:03.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:03.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:03.784+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:03.785+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:03.896+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:03.897808+0000) 2024-02-20T09:21:04.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:04.159+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:04.160527+0000) 2024-02-20T09:21:04.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:04.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:04.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:04.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:04.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:04.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:04.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:04.293+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:04.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:04.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:04.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:04.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:04.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:04.122+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:04.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:04.123+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:04.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:04.161+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:04.162672+0000) 2024-02-20T09:21:05.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:04.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:05.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:04.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:05.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:04.822+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:05.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:04.823+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:05.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:04.899+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:04.900281+0000) 2024-02-20T09:21:05.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:05.161+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:05.162626+0000) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:05.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:05.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:05.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:05.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:05.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:05.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:05.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:05.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:05.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:05.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:05.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:05.164+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:05.165074+0000) 2024-02-20T09:21:06.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:05.809+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:06.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:05.810+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:06.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:05.810+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:06.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:05.811+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:06.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:05.901+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:05.902916+0000) 2024-02-20T09:21:06.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:06.164+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.165062+0000) 2024-02-20T09:21:06.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:06.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:06.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:06.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:06.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:06.337+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:06.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:06.338+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:06.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:06.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:06.414 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:06.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:06.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:06.138+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:06.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:06.139+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:06.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:06.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.167731+0000) 2024-02-20T09:21:07.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:06.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:07.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:06.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:07.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:06.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:07.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:06.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:07.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:06.904+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.904965+0000) 2024-02-20T09:21:07.411 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:07.405+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62406 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:07.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:07.402+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.403438+0000) 2024-02-20T09:21:07.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:07.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:07.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:07.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:07.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:07.156+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:07.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:07.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:07.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:07.169+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.170258+0000) 2024-02-20T09:21:07.413 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:07.402+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.403432+0000) 2024-02-20T09:21:07.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:07.166+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.167021+0000) 2024-02-20T09:21:07.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:07.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:07.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:07.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:07.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:07.324+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:07.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:07.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:07.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:07.402+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.403090+0000) 2024-02-20T09:21:08.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:07.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:08.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:07.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:08.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:07.799+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:08.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:07.800+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:08.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:07.904+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.905266+0000) 2024-02-20T09:21:08.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:08.166+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:08.167331+0000) 2024-02-20T09:21:08.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:08.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:08.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:08.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:08.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:08.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:08.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:08.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:08.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:08.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:08.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:08.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:08.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:08.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:08.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:08.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:08.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:08.169+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:08.170428+0000) 2024-02-20T09:21:09.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:08.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:09.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:08.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:09.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:08.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:09.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:08.841+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:09.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:08.906+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:08.907864+0000) 2024-02-20T09:21:09.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:09.168+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:09.169512+0000) 2024-02-20T09:21:09.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:09.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:09.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:09.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:09.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:09.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:09.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:09.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:09.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:09.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:09.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:09.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:09.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:09.128+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:09.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:09.129+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:09.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:09.172+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:09.173732+0000) 2024-02-20T09:21:10.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:09.839+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:09.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:09.840+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:09.840+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:10.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:09.910+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:09.911438+0000) 2024-02-20T09:21:10.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:10.172+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:10.172954+0000) 2024-02-20T09:21:10.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:10.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:10.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:10.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:10.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:10.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:10.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:10.385+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:10.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:10.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:10.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:10.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:10.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:10.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:10.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:10.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:10.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:10.175+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:10.176250+0000) 2024-02-20T09:21:11.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:10.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:11.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:10.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:11.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:10.857+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:11.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:10.857+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:11.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:10.913+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:10.914824+0000) 2024-02-20T09:21:11.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:11.174+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.175509+0000) 2024-02-20T09:21:11.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:11.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:11.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:11.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:11.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:11.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:11.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:11.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:11.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:11.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:11.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:11.178+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.179652+0000) 2024-02-20T09:21:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:11.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:11.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:11.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:11.421+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:12.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:11.862+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:11.862+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:11.862+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:11.863+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:11.915+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.916530+0000) 2024-02-20T09:21:12.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:12.176+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.177289+0000) 2024-02-20T09:21:12.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:12.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:12.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:12.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:12.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:12.125+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:12.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:12.127+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:12.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:12.180+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.181303+0000) 2024-02-20T09:21:12.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:12.411+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62446 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:12.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:12.409+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.410862+0000) 2024-02-20T09:21:12.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:12.409+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.410316+0000) 2024-02-20T09:21:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:12.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:12.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:12.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:12.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:12.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:12.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:12.409+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.410426+0000) 2024-02-20T09:21:13.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:12.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:13.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:12.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:13.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:12.849+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:13.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:12.850+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:13.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:12.915+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.916717+0000) 2024-02-20T09:21:13.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:13.176+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:13.177482+0000) 2024-02-20T09:21:13.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:13.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:13.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:13.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:13.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:13.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:13.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:13.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:13.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:13.096+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:13.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:13.097+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:13.379 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:13.180+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:13.181518+0000) 2024-02-20T09:21:13.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:13.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:13.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:13.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:13.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:13.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:13.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:13.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:13.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:13.819+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:13.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:13.820+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:14.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:13.919+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:13.920279+0000) 2024-02-20T09:21:14.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:14.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:14.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:14.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:14.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:14.116+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:14.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:14.117+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:14.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:14.184+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:14.185067+0000) 2024-02-20T09:21:14.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:14.178+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:14.179726+0000) 2024-02-20T09:21:14.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:14.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:14.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:14.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:14.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:14.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:14.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:14.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:15.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:14.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:15.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:14.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:15.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:14.863+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:15.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:14.864+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:15.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:14.921+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:14.922512+0000) 2024-02-20T09:21:15.414 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:15.180+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:15.181555+0000) 2024-02-20T09:21:15.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:15.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:15.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:15.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:15.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:15.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:15.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:15.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:15.415 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:15.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:15.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:15.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:15.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:15.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:15.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:15.151+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:15.416 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:15.186+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:15.187149+0000) 2024-02-20T09:21:16.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:15.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:16.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:15.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:16.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:15.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:16.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:15.872+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:16.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:15.923+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:15.924687+0000) 2024-02-20T09:21:16.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:16.182+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.183790+0000) 2024-02-20T09:21:16.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:16.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:16.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:16.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:16.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:16.107+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:16.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:16.108+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:16.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:16.188+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.189342+0000) 2024-02-20T09:21:16.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:16.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:16.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:16.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:16.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:16.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:16.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:17.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:16.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:17.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:16.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:17.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:16.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:17.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:16.901+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:17.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:16.925+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.926348+0000) 2024-02-20T09:21:17.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:17.184+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.185484+0000) 2024-02-20T09:21:17.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:17.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:17.372 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:17.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:17.372 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:17.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:17.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:17.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:17.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:17.101+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:17.373 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:17.190+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.190971+0000) 2024-02-20T09:21:17.636 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:17.420+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62485 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:17.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:17.416+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.417431+0000) 2024-02-20T09:21:17.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:17.415+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.416794+0000) 2024-02-20T09:21:17.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:17.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:17.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:17.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:17.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:17.370+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:17.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:17.415+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.416540+0000) 2024-02-20T09:21:18.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:18.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:18.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:18.088+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:18.089+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:18.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:17.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:18.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:17.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:18.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:17.920+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:17.921+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:17.925+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.926542+0000) 2024-02-20T09:21:18.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:18.190+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:18.191147+0000) 2024-02-20T09:21:18.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:18.184+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:18.185671+0000) 2024-02-20T09:21:18.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:18.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:18.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:18.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:18.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:18.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:18.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:18.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:19.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:18.927+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:18.928719+0000) 2024-02-20T09:21:19.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:18.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:19.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:18.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:19.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:18.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:19.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:18.934+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:19.315 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:19.188+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:19.188977+0000) 2024-02-20T09:21:19.315 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:19.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:19.315 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:19.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:19.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:19.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:19.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:19.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:19.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:19.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:19.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:19.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:19.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:19.047+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:19.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:19.047+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:19.317 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:19.192+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:19.193288+0000) 2024-02-20T09:21:19.887 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:19.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:19.887 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:19.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:19.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:19.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:19.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:19.885+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:20.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:19.931+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:19.932026+0000) 2024-02-20T09:21:20.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:20.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:20.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:20.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:20.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:20.014+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:20.194 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:20.015+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:20.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:20.195+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:20.196882+0000) 2024-02-20T09:21:20.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:20.191+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:20.192557+0000) 2024-02-20T09:21:20.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:20.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:20.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:20.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:20.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:20.295+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:20.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:20.296+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:21.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:21.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.004+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.005+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:20.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:20.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:20.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:20.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:21.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:20.933+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:20.934313+0000) 2024-02-20T09:21:21.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.199+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.200475+0000) 2024-02-20T09:21:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:21.194+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.195397+0000) 2024-02-20T09:21:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:21.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:21.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:21.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:21.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:21.326+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:21.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:21.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:21.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:21.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:21.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:21.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:21.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:21.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:22.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:21.935+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.936417+0000) 2024-02-20T09:21:22.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:22.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:22.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:22.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:21.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:22.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:22.427+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62525 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:22.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:22.425+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.425943+0000) 2024-02-20T09:21:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:22.202+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.203023+0000) 2024-02-20T09:21:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:22.426+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.427547+0000) 2024-02-20T09:21:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:22.196+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.197845+0000) 2024-02-20T09:21:22.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:22.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:22.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:22.288+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:22.288+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:22.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:22.425+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.426141+0000) 2024-02-20T09:21:23.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:22.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:23.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:22.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:23.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:22.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:23.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:22.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:23.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:22.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:23.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:22.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:23.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:22.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:23.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:22.883+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:23.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:22.935+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.936876+0000) 2024-02-20T09:21:23.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:23.202+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:23.203199+0000) 2024-02-20T09:21:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:23.197+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:23.198052+0000) 2024-02-20T09:21:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:23.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:23.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:23.336+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:23.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:23.337+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:23.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:23.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:23.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:23.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:23.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:23.869+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:23.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:23.870+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:24.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:23.938+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:23.938933+0000) 2024-02-20T09:21:24.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:23.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:24.201 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:23.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:24.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:23.958+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:24.202 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:23.959+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:24.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:24.204+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:24.205190+0000) 2024-02-20T09:21:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:24.199+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:24.200155+0000) 2024-02-20T09:21:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:24.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:24.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:24.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:24.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:24.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:24.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:24.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:24.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:25.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:24.947+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:24.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:24.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:24.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:24.912+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:25.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:24.941+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:24.942564+0000) 2024-02-20T09:21:25.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:25.207+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:25.208533+0000) 2024-02-20T09:21:25.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:25.202+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:25.203431+0000) 2024-02-20T09:21:25.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:25.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:25.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:25.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:25.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:25.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:25.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:25.306+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:26.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:25.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:26.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:25.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:26.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:25.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:26.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:25.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:26.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:25.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:26.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:25.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:26.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:25.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:26.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:25.925+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:26.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:25.943+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:25.944769+0000) 2024-02-20T09:21:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:26.210+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.211711+0000) 2024-02-20T09:21:26.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:26.206+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.206954+0000) 2024-02-20T09:21:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:26.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:26.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:26.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:26.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:27.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:26.945+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.946343+0000) 2024-02-20T09:21:27.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:26.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:27.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:26.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:27.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:26.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:27.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:26.958+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:27.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:26.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:27.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:26.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:27.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:26.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:27.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:26.930+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:27.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:27.435+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62564 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:27.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:27.432+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.433096+0000) 2024-02-20T09:21:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:27.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.213691+0000) 2024-02-20T09:21:27.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:27.433+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.433983+0000) 2024-02-20T09:21:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:27.208+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.209153+0000) 2024-02-20T09:21:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:27.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:27.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:27.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:27.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:27.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:27.432+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.433794+0000) 2024-02-20T09:21:28.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:27.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:28.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:27.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:28.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:27.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:28.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:27.948+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:27.945+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.946457+0000) 2024-02-20T09:21:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:27.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:27.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:27.974+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:28.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:27.974+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:28.212+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:28.213882+0000) 2024-02-20T09:21:28.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:28.208+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:28.209352+0000) 2024-02-20T09:21:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:28.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:28.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:28.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:28.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:29.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:28.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:29.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:28.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:29.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:28.923+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:29.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:28.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:29.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:28.947+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:28.948634+0000) 2024-02-20T09:21:29.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:28.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:29.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:28.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:29.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:28.950+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:29.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:28.951+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:29.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:29.214+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:29.215925+0000) 2024-02-20T09:21:29.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:29.211+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:29.211975+0000) 2024-02-20T09:21:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:29.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:29.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:29.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:29.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:29.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:30.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:29.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:30.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:29.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:30.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:29.940+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:30.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:29.941+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:30.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:29.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:30.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:29.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:30.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:29.910+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:30.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:29.911+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:30.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:29.949+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:29.950730+0000) 2024-02-20T09:21:30.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:30.217+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:30.218532+0000) 2024-02-20T09:21:30.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:30.214+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:30.215495+0000) 2024-02-20T09:21:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:30.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:30.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:30.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:30.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:30.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:30.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:30.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:30.914 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:30.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:30.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:30.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:30.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:30.876+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:31.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:30.953+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:30.954080+0000) 2024-02-20T09:21:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:30.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:30.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:31.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:30.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:31.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:30.913+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:31.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:31.219+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.220705+0000) 2024-02-20T09:21:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:31.217+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.218135+0000) 2024-02-20T09:21:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:31.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:31.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:31.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:31.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:31.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:31.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:31.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:31.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:31.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:31.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:31.879+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:31.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:31.879+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:32.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:31.955+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.956574+0000) 2024-02-20T09:21:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:31.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:31.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:31.961+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:31.962+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:32.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:32.440+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.441261+0000) 2024-02-20T09:21:32.472 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:32.444+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62604 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:32.222+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.223452+0000) 2024-02-20T09:21:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:32.440+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.441300+0000) 2024-02-20T09:21:32.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:32.218+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.219639+0000) 2024-02-20T09:21:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:32.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:32.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:32.317+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:32.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:32.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:32.439+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.440773+0000) 2024-02-20T09:21:33.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:32.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:32.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:32.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:33.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:32.916+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:33.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:32.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:33.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:32.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:32.852+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:32.853+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:32.955+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.956749+0000) 2024-02-20T09:21:33.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:33.222+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:33.223611+0000) 2024-02-20T09:21:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:33.218+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:33.219832+0000) 2024-02-20T09:21:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:33.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:33.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:33.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:33.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:33.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:33.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:33.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:33.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:33.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:33.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:33.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:33.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:33.826+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:34.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:33.959+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:33.960291+0000) 2024-02-20T09:21:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:33.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:33.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:33.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:33.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:34.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:34.224+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:34.225696+0000) 2024-02-20T09:21:34.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:34.222+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:34.223195+0000) 2024-02-20T09:21:34.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:34.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:34.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:34.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:34.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:34.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:34.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:34.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:34.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:34.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:34.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:34.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:34.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:34.858+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:34.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:34.859+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:34.962+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:34.962953+0000) 2024-02-20T09:21:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:34.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:35.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:34.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:34.949+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:34.951+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:35.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:35.228+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:35.229256+0000) 2024-02-20T09:21:35.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:35.224+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:35.225322+0000) 2024-02-20T09:21:35.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:35.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:35.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:35.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:35.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:35.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:35.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:35.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:35.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:35.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:35.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:35.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:35.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:35.834+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:35.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:35.835+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:36.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:35.965+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:35.966245+0000) 2024-02-20T09:21:36.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:35.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:35.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:35.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:36.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:35.997+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:36.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:36.227+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.228898+0000) 2024-02-20T09:21:36.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:36.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:36.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:36.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:36.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:36.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:36.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:36.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:36.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:36.230+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.231425+0000) 2024-02-20T09:21:36.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:36.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:36.954 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:36.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:36.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:36.792+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:36.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:36.793+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:37.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:36.967+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.968032+0000) 2024-02-20T09:21:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:36.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:36.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:36.952+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:36.953+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:37.515 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:37.452+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62643 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:37.449+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.450569+0000) 2024-02-20T09:21:37.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:37.232+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.233482+0000) 2024-02-20T09:21:37.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:37.450+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.450993+0000) 2024-02-20T09:21:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:37.230+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.231539+0000) 2024-02-20T09:21:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:37.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:37.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:37.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:37.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:37.309+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:37.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:37.449+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.450080+0000) 2024-02-20T09:21:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:37.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:37.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:37.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:37.914+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:37.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:37.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:37.827+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:37.829+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:38.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:37.967+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.968194+0000) 2024-02-20T09:21:38.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:38.233+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:38.233965+0000) 2024-02-20T09:21:38.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:38.230+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:38.231728+0000) 2024-02-20T09:21:38.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:38.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:38.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:38.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:38.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:38.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:38.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:38.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:38.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:38.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:38.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:38.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:38.869+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:38.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:38.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:38.824+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:38.825+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:38.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:38.969+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:38.970616+0000) 2024-02-20T09:21:39.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:39.236+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:39.237522+0000) 2024-02-20T09:21:39.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:39.233+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:39.234266+0000) 2024-02-20T09:21:39.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:39.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:39.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:39.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:39.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:39.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:39.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:39.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:39.838+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:39.838+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:39.838+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:39.839+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:39.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:39.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:39.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:39.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:39.861+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:40.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:39.972+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:39.973190+0000) 2024-02-20T09:21:40.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:40.238+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:40.239817+0000) 2024-02-20T09:21:40.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:40.236+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:40.237760+0000) 2024-02-20T09:21:40.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:40.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:40.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:40.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:40.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:40.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:40.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:40.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:41.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:40.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:40.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:40.897+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:40.897+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:40.975+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:40.976745+0000) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:40.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:40.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:40.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:40.863+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:41.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:41.242+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.243024+0000) 2024-02-20T09:21:41.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:41.239+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.240490+0000) 2024-02-20T09:21:41.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:41.304+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:41.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:41.304+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:41.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:41.304+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:41.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:41.305+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:41.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:41.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:41.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:41.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:41.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:41.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:41.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:41.874+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:41.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:41.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:41.929+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:41.930+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:41.977+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.978740+0000) 2024-02-20T09:21:42.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:42.243+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.244920+0000) 2024-02-20T09:21:42.580 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:42.456+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.457673+0000) 2024-02-20T09:21:42.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:42.457+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.458132+0000) 2024-02-20T09:21:42.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:42.459+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62683 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:42.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:42.241+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.242504+0000) 2024-02-20T09:21:42.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:42.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:42.581 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:42.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:42.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:42.281+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:42.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:42.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:42.582 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:42.456+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.457540+0000) 2024-02-20T09:21:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:42.850+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:43.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:42.850+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:42.850+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:42.851+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:42.935+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:42.935+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:42.935+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:42.936+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:43.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:42.978+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.978969+0000) 2024-02-20T09:21:43.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:43.244+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:43.245119+0000) 2024-02-20T09:21:43.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:43.241+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:43.242677+0000) 2024-02-20T09:21:43.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:43.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:43.615 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:43.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:43.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:43.289+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:43.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:43.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:43.893 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:43.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:43.894 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:43.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:43.894 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:43.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:43.894 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:43.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:44.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:43.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:43.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:43.891+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:43.893+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:43.981+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:43.982364+0000) 2024-02-20T09:21:44.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:44.246+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:44.247567+0000) 2024-02-20T09:21:44.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:44.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:44.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:44.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:44.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:44.243+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:44.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:44.244+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:44.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:44.244+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:44.245866+0000) 2024-02-20T09:21:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:44.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:44.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:44.862+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:44.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:44.863+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:44.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:44.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:44.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:44.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:44.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:44.868+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:44.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:44.869+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:45.249 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:44.984+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:44.985651+0000) 2024-02-20T09:21:45.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:45.249+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:45.250078+0000) 2024-02-20T09:21:45.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:45.247+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:45.248585+0000) 2024-02-20T09:21:45.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:45.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:45.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:45.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:45.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:45.253+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:45.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:45.254+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:45.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:45.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:45.826+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:45.827+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:45.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:45.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:45.842+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:45.842+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:46.252 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:45.988+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:45.988978+0000) 2024-02-20T09:21:46.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:46.252+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.253519+0000) 2024-02-20T09:21:46.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:46.251+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.251977+0000) 2024-02-20T09:21:46.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:46.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:46.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:46.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:46.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:46.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:46.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:46.279+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:46.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:46.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:46.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:46.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:46.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:46.873+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:46.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:46.874+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:47.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:46.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:46.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:46.889+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:46.890+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:47.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:46.990+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.991739+0000) 2024-02-20T09:21:47.640 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:47.467+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62722 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:47.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:47.254+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.255287+0000) 2024-02-20T09:21:47.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:47.464+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.465591+0000) 2024-02-20T09:21:47.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:47.464+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.465497+0000) 2024-02-20T09:21:47.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:47.252+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.253877+0000) 2024-02-20T09:21:47.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:47.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:47.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:47.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:47.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:47.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:47.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:47.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:47.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:47.464+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.465918+0000) 2024-02-20T09:21:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:47.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:47.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:47.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:47.885+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:47.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:47.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:47.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:47.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:48.254 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:47.991+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.991934+0000) 2024-02-20T09:21:48.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:48.254+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:48.255477+0000) 2024-02-20T09:21:48.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:48.253+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:48.254014+0000) 2024-02-20T09:21:48.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:48.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:48.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:48.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:48.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:48.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:48.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:48.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:48.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:48.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:48.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:48.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:48.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:48.871+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:48.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:48.872+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:49.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:48.993+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:48.994031+0000) 2024-02-20T09:21:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:48.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:48.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:48.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:48.901+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:49.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:49.255+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:49.256650+0000) 2024-02-20T09:21:49.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:49.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:49.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:49.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:49.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:49.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:49.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:49.267+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:49.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:49.257+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:49.258724+0000) 2024-02-20T09:21:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:49.882+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:49.882+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:49.882+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:49.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:49.883+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:49.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:49.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:49.882+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:49.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:49.883+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:50.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:49.995+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:49.996459+0000) 2024-02-20T09:21:50.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:50.258+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:50.259790+0000) 2024-02-20T09:21:50.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:50.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:50.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:50.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:50.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:50.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:50.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:50.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:50.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:50.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:50.260964+0000) 2024-02-20T09:21:50.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:50.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:50.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:50.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:50.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:50.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:50.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:50.867+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:50.997+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:50.998904+0000) 2024-02-20T09:21:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:50.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:50.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:50.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:50.915+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:51.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:51.261+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.262226+0000) 2024-02-20T09:21:51.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:51.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:51.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:51.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:51.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:51.312+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:51.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:51.313+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:51.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:51.263+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.264081+0000) 2024-02-20T09:21:51.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:51.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:51.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:51.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:51.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:51.825+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:51.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:51.826+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:52.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.000+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.000990+0000) 2024-02-20T09:21:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:51.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:51.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:51.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:51.916+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:52.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:52.474+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62762 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:52.597 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.472+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.474924+0000) 2024-02-20T09:21:52.597 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:52.264+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.266406+0000) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:52.471+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.473361+0000) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:52.262+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.264468+0000) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:52.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:52.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:52.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:52.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:52.598 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:52.472+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.474526+0000) 2024-02-20T09:21:52.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:52.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:52.893 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:52.893 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:53.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:52.999+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:53.001191+0000) 2024-02-20T09:21:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:52.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:52.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:52.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:52.890+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:53.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:53.262+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:53.264706+0000) 2024-02-20T09:21:53.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:53.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:53.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:53.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:53.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:53.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:53.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:53.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:53.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:53.264+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:53.266561+0000) 2024-02-20T09:21:53.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:53.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:53.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:53.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:53.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:53.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:53.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:53.829+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:54.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:54.001+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:54.003818+0000) 2024-02-20T09:21:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:53.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:53.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:53.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:54.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:53.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:54.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:54.267+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:54.269027+0000) 2024-02-20T09:21:54.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:54.265+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:54.267225+0000) 2024-02-20T09:21:54.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:54.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:54.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:54.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:54.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:54.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:54.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:54.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:54.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:54.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:54.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:54.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:54.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:54.813+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:54.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:54.814+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:55.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:55.004+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:55.005902+0000) 2024-02-20T09:21:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:54.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:54.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:54.928+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:54.928+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:55.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:55.269+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:55.271763+0000) 2024-02-20T09:21:55.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:55.269+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:55.270304+0000) 2024-02-20T09:21:55.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:55.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:55.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:55.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:55.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:55.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:55.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:55.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:55.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:55.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:55.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:55.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:55.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:55.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:55.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:55.774+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:56.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:56.007+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.008813+0000) 2024-02-20T09:21:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:55.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:55.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:55.902+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:55.903+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:56.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:56.273+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.274473+0000) 2024-02-20T09:21:56.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:56.272+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.273154+0000) 2024-02-20T09:21:56.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:56.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:56.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:56.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:56.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:56.351+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:56.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:56.352+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:56.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:56.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:56.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:56.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:56.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:56.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:56.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:56.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:56.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:56.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:56.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:56.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:56.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:56.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:56.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:56.767+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:57.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:57.009+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.010712+0000) 2024-02-20T09:21:57.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:57.275+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.276196+0000) 2024-02-20T09:21:57.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:57.480+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.481782+0000) 2024-02-20T09:21:57.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:21:57.482+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62801 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:21:57.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:57.274+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.275117+0000) 2024-02-20T09:21:57.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:57.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:57.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:57.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:57.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:57.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:57.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:57.382+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:57.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:57.480+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.481585+0000) 2024-02-20T09:21:57.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:57.480+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.481532+0000) 2024-02-20T09:21:57.883 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:57.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:57.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:57.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:57.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:57.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:57.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:57.788+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:58.010+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:58.010933+0000) 2024-02-20T09:21:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:57.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:57.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:57.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:57.882+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:58.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:58.274+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:58.275291+0000) 2024-02-20T09:21:58.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:58.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:58.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:58.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:58.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:58.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:58.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:58.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:58.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:58.275+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:58.276400+0000) 2024-02-20T09:21:58.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:58.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:58.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:58.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:58.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:58.818+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:58.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:58.818+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:59.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:59.012+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:59.012956+0000) 2024-02-20T09:21:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:58.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:58.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:58.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:21:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:58.922+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:59.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:59.278+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:59.279070+0000) 2024-02-20T09:21:59.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:59.277+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:59.278338+0000) 2024-02-20T09:21:59.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:59.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:59.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:59.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:59.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:59.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:21:59.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:21:59.446+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:21:59.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:59.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:59.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:59.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:59.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:59.853+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:21:59.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:21:59.854+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:00.014+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:00.015343+0000) 2024-02-20T09:22:00.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:59.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:59.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:59.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:21:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:21:59.951+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:00.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:00.280+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:00.281749+0000) 2024-02-20T09:22:00.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:00.280+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:00.281689+0000) 2024-02-20T09:22:00.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:00.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:00.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:00.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:00.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:00.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:00.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:00.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:00.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:00.888+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:00.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:00.888+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:00.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:00.888+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:00.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:00.889+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:01.017+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.017928+0000) 2024-02-20T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:00.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:00.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:00.909+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:00.910+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:01.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:01.283+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.284812+0000) 2024-02-20T09:22:01.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:01.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:01.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:01.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:01.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:01.461+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:01.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:01.462+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:01.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:01.283+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.284615+0000) 2024-02-20T09:22:01.900 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:01.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:01.900 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:01.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:01.900 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:01.872+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:01.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:01.873+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:02.019+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.019955+0000) 2024-02-20T09:22:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:01.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:01.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:01.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:01.899+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:02.491+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62841 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:02.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:02.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.287141+0000) 2024-02-20T09:22:02.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:02.486+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.487870+0000) 2024-02-20T09:22:02.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:02.285+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.286763+0000) 2024-02-20T09:22:02.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:02.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:02.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:02.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:02.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:02.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:02.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:02.450+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:02.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:02.487+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.488516+0000) 2024-02-20T09:22:02.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:02.487+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.487981+0000) 2024-02-20T09:22:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:02.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:02.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:02.915+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:02.916+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:02.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:02.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:02.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:02.899+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:03.019+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:03.020148+0000) 2024-02-20T09:22:03.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:03.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:03.287322+0000) 2024-02-20T09:22:03.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:03.286+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:03.286952+0000) 2024-02-20T09:22:03.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:03.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:03.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:03.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:03.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:03.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:03.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:03.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:03.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:03.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:03.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:03.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:03.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:03.881+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:03.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:03.882+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:04.021+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:04.022293+0000) 2024-02-20T09:22:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:03.916+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:03.916+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:03.916+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:04.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:03.917+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:04.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:04.288+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:04.289740+0000) 2024-02-20T09:22:04.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:04.289+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:04.290103+0000) 2024-02-20T09:22:04.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:04.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:04.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:04.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:04.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:04.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:04.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:04.454+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:05.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:04.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:05.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:04.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:04.917+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:04.918+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:04.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:04.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:04.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:04.901+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:05.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:05.024+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:05.025445+0000) 2024-02-20T09:22:05.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:05.291+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:05.292612+0000) 2024-02-20T09:22:05.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:05.291+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:05.292900+0000) 2024-02-20T09:22:05.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:05.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:05.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:05.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:05.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:05.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:05.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:05.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:05.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:05.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:05.950+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:06.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:05.950+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:05.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:05.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:05.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:05.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:06.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:06.026+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.027733+0000) 2024-02-20T09:22:06.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:06.294+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.295891+0000) 2024-02-20T09:22:06.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:06.294+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.295804+0000) 2024-02-20T09:22:06.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:06.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:06.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:06.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:06.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:06.406+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:06.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:06.407+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:07.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:06.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:06.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:06.978+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:06.978+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:06.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:06.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:07.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:06.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:07.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:06.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:07.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:07.028+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.029757+0000) 2024-02-20T09:22:07.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:07.297+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.298686+0000) 2024-02-20T09:22:07.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:07.496+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.496944+0000) 2024-02-20T09:22:07.622 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:07.499+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62880 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:07.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:07.296+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.297845+0000) 2024-02-20T09:22:07.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:07.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:07.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:07.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:07.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:07.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:07.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:07.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:07.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:07.496+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.497822+0000) 2024-02-20T09:22:07.623 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:07.495+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.496320+0000) 2024-02-20T09:22:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:07.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:07.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:07.986+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:07.987+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.005+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:08.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.029+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:08.029957+0000) 2024-02-20T09:22:08.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:08.297+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:08.298882+0000) 2024-02-20T09:22:08.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:08.297+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:08.298018+0000) 2024-02-20T09:22:08.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:08.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:08.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:08.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:08.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:08.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:08.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:08.366+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.023+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.024+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:09.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:08.960+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:09.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:09.030+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:09.031754+0000) 2024-02-20T09:22:09.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.300+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:09.301660+0000) 2024-02-20T09:22:09.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:09.300+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:09.301102+0000) 2024-02-20T09:22:09.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:09.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:09.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:09.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:09.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:09.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:09.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:09.320+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:10.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:10.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:10.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:10.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:09.996+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:10.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:09.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:10.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:09.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:10.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:09.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:10.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:09.999+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:10.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:10.034+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:10.035154+0000) 2024-02-20T09:22:10.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:10.304+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:10.305158+0000) 2024-02-20T09:22:10.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:10.303+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:10.303954+0000) 2024-02-20T09:22:10.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:10.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:10.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:10.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:10.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:10.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:10.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:10.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:11.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:11.008+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:11.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:11.008+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:11.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:11.008+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:11.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:11.009+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:11.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:11.037+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.038623+0000) 2024-02-20T09:22:11.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:11.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:11.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:11.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:11.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:11.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:11.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:11.050+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:11.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:11.307+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.308639+0000) 2024-02-20T09:22:11.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:11.306+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.307439+0000) 2024-02-20T09:22:11.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:11.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:11.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:11.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:11.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:11.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:11.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:11.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:12.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:12.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:12.022+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:12.023+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:12.039+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.040829+0000) 2024-02-20T09:22:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:12.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:12.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:12.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:12.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:12.047+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:12.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:12.309+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.310557+0000) 2024-02-20T09:22:12.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:12.502+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.503912+0000) 2024-02-20T09:22:12.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:12.308+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.309373+0000) 2024-02-20T09:22:12.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:12.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:12.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:12.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:12.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:12.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:12.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:12.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:12.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:12.503+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.504316+0000) 2024-02-20T09:22:12.631 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:12.507+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62921 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:12.631 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:12.503+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.504698+0000) 2024-02-20T09:22:13.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:13.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:13.310 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:13.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:13.007+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:13.008+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.004+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.005+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:13.311 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.040+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:13.041111+0000) 2024-02-20T09:22:13.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:13.309+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:13.310776+0000) 2024-02-20T09:22:13.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:13.308+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:13.309532+0000) 2024-02-20T09:22:13.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:13.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:13.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:13.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:13.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:13.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:13.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:13.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:14.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:14.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:14.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:14.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:14.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:14.055+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:14.313 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:14.056+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:14.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:14.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:14.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.998+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:14.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:13.999+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:14.314 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:14.043+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:14.044475+0000) 2024-02-20T09:22:14.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:14.312+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:14.313333+0000) 2024-02-20T09:22:14.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:14.311+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:14.312235+0000) 2024-02-20T09:22:14.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:14.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:14.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:14.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:14.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:14.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:14.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:14.401+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:15.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:15.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:15.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:15.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:15.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:15.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:15.316 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:15.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:15.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:15.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:15.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:15.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:15.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:15.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:15.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:15.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:15.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:15.046+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:15.047154+0000) 2024-02-20T09:22:15.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:15.315+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:15.316805+0000) 2024-02-20T09:22:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:15.314+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:15.315275+0000) 2024-02-20T09:22:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:15.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:15.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:15.449+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:15.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:15.450+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:16.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:16.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:16.319 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:16.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:16.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:16.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:16.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:16.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:16.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:16.048+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.049257+0000) 2024-02-20T09:22:16.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:16.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:16.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:16.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:16.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:16.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:16.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:16.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:16.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:16.319+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.320148+0000) 2024-02-20T09:22:16.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:16.317+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.318498+0000) 2024-02-20T09:22:16.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:16.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:16.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:16.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:16.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:16.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:16.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:16.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:17.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:17.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:17.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:17.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:17.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:17.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:17.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:17.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:17.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:17.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:17.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:17.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.032+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:17.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.050+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.051566+0000) 2024-02-20T09:22:17.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.512+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.513226+0000) 2024-02-20T09:22:17.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:17.321+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.322563+0000) 2024-02-20T09:22:17.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:17.512+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.513028+0000) 2024-02-20T09:22:17.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:17.320+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.321248+0000) 2024-02-20T09:22:17.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:17.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:17.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:17.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:17.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:17.385+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:17.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:17.386+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:17.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:17.512+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.512971+0000) 2024-02-20T09:22:17.641 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:17.517+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 62960 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:18.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:18.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:18.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.983+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:18.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:17.984+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:18.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:18.050+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:18.051738+0000) 2024-02-20T09:22:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:18.109+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:18.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:18.110+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:18.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:18.321+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:18.322752+0000) 2024-02-20T09:22:18.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:18.320+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:18.321446+0000) 2024-02-20T09:22:18.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:18.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:18.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:18.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:18.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:18.414+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:18.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:18.415+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:19.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:19.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:19.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:19.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:19.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:19.133+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:19.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:19.134+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:19.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:19.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:19.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:19.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:19.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:19.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:19.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:19.026+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:19.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:19.054+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:19.055054+0000) 2024-02-20T09:22:19.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:19.324+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:19.325697+0000) 2024-02-20T09:22:19.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:19.323+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:19.324657+0000) 2024-02-20T09:22:19.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:19.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:19.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:19.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:19.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:19.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:19.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:19.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:20.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:20.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:20.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:20.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:20.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:20.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:20.322 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:20.107+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:20.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:20.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:20.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.000+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:20.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.000+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:20.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.056+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:20.057261+0000) 2024-02-20T09:22:20.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:20.327+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:20.328613+0000) 2024-02-20T09:22:20.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:20.326+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:20.327750+0000) 2024-02-20T09:22:20.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:20.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:20.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:20.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:20.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:20.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:20.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:20.348+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:21.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:21.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:21.106+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:21.107+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.960+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:20.961+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:21.059+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.060071+0000) 2024-02-20T09:22:21.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:21.330+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.331916+0000) 2024-02-20T09:22:21.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:21.329+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.330623+0000) 2024-02-20T09:22:21.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:21.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:21.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:21.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:21.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:21.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:21.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:21.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:22.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:22.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:22.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:22.064+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:22.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:22.065+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:21.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:21.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:21.934+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:21.935+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:22.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:22.060+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.061879+0000) 2024-02-20T09:22:22.643 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:22.524+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63000 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:22.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:22.331+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.332592+0000) 2024-02-20T09:22:22.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:22.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:22.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:22.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:22.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:22.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:22.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:22.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:22.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:22.521+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.522607+0000) 2024-02-20T09:22:22.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:22.523+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.524417+0000) 2024-02-20T09:22:22.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:22.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.334130+0000) 2024-02-20T09:22:22.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:22.521+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.522397+0000) 2024-02-20T09:22:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:23.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:23.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:23.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:23.105+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:23.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:23.106+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:22.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:22.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:22.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:23.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:22.966+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:23.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:23.061+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:23.062120+0000) 2024-02-20T09:22:23.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:23.331+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:23.332792+0000) 2024-02-20T09:22:23.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:23.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:23.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:23.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:23.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:23.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:23.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:23.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:23.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:23.333+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:23.334314+0000) 2024-02-20T09:22:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:24.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:24.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:24.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:24.079+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:24.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:24.080+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:23.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:23.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:23.949+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:23.950+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:24.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:24.064+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:24.065665+0000) 2024-02-20T09:22:24.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:24.334+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:24.335215+0000) 2024-02-20T09:22:24.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:24.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:24.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:24.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:24.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:24.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:24.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:24.359+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:24.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:24.336+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:24.337697+0000) 2024-02-20T09:22:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:25.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:25.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:25.042+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:25.042+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:24.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:24.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:24.953+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:24.954+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:25.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:25.067+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:25.068397+0000) 2024-02-20T09:22:25.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:25.339+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:25.339951+0000) 2024-02-20T09:22:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:25.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:25.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:25.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:25.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:25.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:25.337+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:25.338775+0000) 2024-02-20T09:22:26.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:26.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:26.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:26.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:26.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:26.024+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:26.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:26.026+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:26.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:25.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:26.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:25.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:26.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:25.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:26.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:25.988+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:26.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:26.070+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.071911+0000) 2024-02-20T09:22:26.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:26.341+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.342524+0000) 2024-02-20T09:22:26.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:26.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:26.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:26.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:26.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:26.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:26.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:26.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:26.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:26.340+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.341287+0000) 2024-02-20T09:22:27.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:27.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:27.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:27.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:27.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:27.052+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:27.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:27.053+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:27.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:27.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:27.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:27.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:27.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:27.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:27.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:27.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:27.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:27.073+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.074495+0000) 2024-02-20T09:22:27.643 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:27.533+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63039 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:27.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:27.529+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.530727+0000) 2024-02-20T09:22:27.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:27.343+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.344753+0000) 2024-02-20T09:22:27.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:27.529+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.530300+0000) 2024-02-20T09:22:27.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:27.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:27.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:27.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:27.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:27.339+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:27.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:27.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:27.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:27.342+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.343507+0000) 2024-02-20T09:22:27.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:27.530+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.531202+0000) 2024-02-20T09:22:28.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:28.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:28.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:28.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:28.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:28.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:28.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:28.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:28.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:28.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:28.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:28.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:28.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:28.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:28.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:28.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:28.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:28.074+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:28.074953+0000) 2024-02-20T09:22:28.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:28.344+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:28.344950+0000) 2024-02-20T09:22:28.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:28.342+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:28.343698+0000) 2024-02-20T09:22:28.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:28.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:28.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:28.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:28.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:28.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:28.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:28.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:29.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:29.031+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:29.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:29.031+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:29.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:29.031+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:29.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:29.032+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:29.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:29.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:29.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:29.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:29.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:29.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:29.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:29.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:29.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:29.077+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:29.078309+0000) 2024-02-20T09:22:29.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:29.347+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:29.347959+0000) 2024-02-20T09:22:29.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:29.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:29.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:29.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:29.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:29.335+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:29.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:29.336+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:29.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:29.346+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:29.347221+0000) 2024-02-20T09:22:30.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.008+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:30.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.008+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:30.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.008+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:30.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.010+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:30.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:30.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:30.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:30.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:30.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.080+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:30.081573+0000) 2024-02-20T09:22:30.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.350+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:30.351820+0000) 2024-02-20T09:22:30.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:30.349+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:30.350564+0000) 2024-02-20T09:22:30.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:30.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:30.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:30.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:30.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:30.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:30.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:30.354+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:31.323 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:31.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:31.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:31.324 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:30.976+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:31.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:31.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:31.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.996+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:31.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:30.997+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:31.325 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:31.082+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.083851+0000) 2024-02-20T09:22:31.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:31.353+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.354925+0000) 2024-02-20T09:22:31.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:31.353+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.354222+0000) 2024-02-20T09:22:31.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:31.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:31.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:31.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:31.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:31.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:31.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:31.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:31.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:31.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:31.930+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:31.932+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:32.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:32.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:32.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:32.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:32.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:32.085+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.086679+0000) 2024-02-20T09:22:32.644 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:32.540+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63079 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:32.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:32.538+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.539392+0000) 2024-02-20T09:22:32.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:32.356+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.357660+0000) 2024-02-20T09:22:32.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:32.537+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.538427+0000) 2024-02-20T09:22:32.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:32.356+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.356954+0000) 2024-02-20T09:22:32.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:32.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:32.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:32.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:32.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:32.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:32.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:32.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:32.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:32.538+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.539435+0000) 2024-02-20T09:22:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:32.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:32.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:32.887+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:32.888+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:33.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:33.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:33.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:33.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:33.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:33.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:33.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:33.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:33.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:33.085+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:33.086871+0000) 2024-02-20T09:22:33.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:33.356+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:33.357856+0000) 2024-02-20T09:22:33.644 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:33.356+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:33.357154+0000) 2024-02-20T09:22:33.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:33.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:33.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:33.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:33.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:33.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:33.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:33.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:33.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:33.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:33.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:33.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:34.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:34.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:34.073+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:34.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:34.088+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:34.089257+0000) 2024-02-20T09:22:34.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:34.359+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:34.360896+0000) 2024-02-20T09:22:34.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:34.345+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:34.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:34.345+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:34.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:34.345+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:34.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:34.346+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:34.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:34.358+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:34.359572+0000) 2024-02-20T09:22:34.894 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:34.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:34.895 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:34.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:34.895 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:34.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:34.895 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:34.893+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:35.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:35.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:35.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:35.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:35.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:35.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:35.090+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:35.091647+0000) 2024-02-20T09:22:35.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:35.363+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:35.364649+0000) 2024-02-20T09:22:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:35.361+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:35.362478+0000) 2024-02-20T09:22:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:35.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:35.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:35.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:35.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:35.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:35.848+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:35.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:35.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:35.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:36.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:36.092+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.093721+0000) 2024-02-20T09:22:36.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:36.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:36.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:36.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:36.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:36.094+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:36.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:36.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:36.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:36.367+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.368427+0000) 2024-02-20T09:22:36.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:36.364+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.365866+0000) 2024-02-20T09:22:36.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:36.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:36.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:36.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:36.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:36.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:36.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:36.389+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:36.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:36.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:36.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:36.885+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:37.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:37.094+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.095795+0000) 2024-02-20T09:22:37.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:37.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:37.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:37.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:37.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:37.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:37.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:37.136+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:37.644 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:37.549+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63118 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:37.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:37.545+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.546097+0000) 2024-02-20T09:22:37.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:37.370+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.371300+0000) 2024-02-20T09:22:37.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:37.545+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.546858+0000) 2024-02-20T09:22:37.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:37.367+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.368536+0000) 2024-02-20T09:22:37.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:37.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:37.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:37.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:37.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:37.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:37.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:37.431+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:37.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:37.545+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.546250+0000) 2024-02-20T09:22:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:37.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:37.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:37.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:37.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:38.095+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:38.095954+0000) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:38.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:38.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:38.138+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:38.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:38.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:38.370+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:38.371475+0000) 2024-02-20T09:22:38.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:38.367+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:38.368743+0000) 2024-02-20T09:22:38.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:38.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:38.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:38.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:38.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:38.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:38.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:38.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:38.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:38.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:38.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:38.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:38.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:38.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:39.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:38.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:39.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:39.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:39.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:39.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:39.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:39.098+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:39.099261+0000) 2024-02-20T09:22:39.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:39.373+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:39.373956+0000) 2024-02-20T09:22:39.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:39.370+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:39.371187+0000) 2024-02-20T09:22:39.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:39.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:39.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:39.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:39.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:39.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:39.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:39.480+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:39.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:39.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:39.906+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:39.907+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:40.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:40.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:40.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:40.070+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:40.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:40.101+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:40.102535+0000) 2024-02-20T09:22:40.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:40.375+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:40.376593+0000) 2024-02-20T09:22:40.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:40.372+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:40.373620+0000) 2024-02-20T09:22:40.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:40.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:40.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:40.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:40.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:40.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:40.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:40.471+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:41.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:40.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:40.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:40.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:40.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:41.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:41.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:41.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:41.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:41.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:41.104+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.105130+0000) 2024-02-20T09:22:41.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:41.378+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.379589+0000) 2024-02-20T09:22:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:41.376+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.377015+0000) 2024-02-20T09:22:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:41.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:41.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:41.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:41.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:41.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:41.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:41.508+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:41.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:41.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:41.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:41.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:42.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:42.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:42.078+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:42.079+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:42.106+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.107862+0000) 2024-02-20T09:22:42.576 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:42.556+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63158 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:42.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:42.380+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.381340+0000) 2024-02-20T09:22:42.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:42.558+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.559273+0000) 2024-02-20T09:22:42.577 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:42.553+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.554591+0000) 2024-02-20T09:22:42.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:42.378+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.379091+0000) 2024-02-20T09:22:42.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:42.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:42.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:42.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:42.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:42.491+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:42.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:42.492+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:42.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:42.553+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.554556+0000) 2024-02-20T09:22:43.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:42.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:43.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:42.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:43.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:42.981+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:43.109 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:42.982+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:43.109 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:43.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:43.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:43.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:43.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:43.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:43.110 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:43.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:43.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:43.107+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:43.108051+0000) 2024-02-20T09:22:43.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:43.380+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:43.381498+0000) 2024-02-20T09:22:43.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:43.378+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:43.379308+0000) 2024-02-20T09:22:43.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:43.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:43.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:43.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:43.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:43.485+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:43.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:43.486+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:43.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:43.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:43.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:43.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:44.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:44.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:44.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:44.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:44.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:44.109+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:44.110498+0000) 2024-02-20T09:22:44.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:44.383+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:44.384752+0000) 2024-02-20T09:22:44.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:44.381+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:44.381978+0000) 2024-02-20T09:22:44.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:44.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:44.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:44.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:44.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:44.498+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:44.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:44.499+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:44.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:44.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:44.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:44.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:45.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:45.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:45.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:45.112+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:45.113065+0000) 2024-02-20T09:22:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:45.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:45.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:45.387+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:45.388271+0000) 2024-02-20T09:22:45.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:45.384+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:45.385317+0000) 2024-02-20T09:22:45.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:45.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:45.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:45.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:45.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:45.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:45.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:45.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:45.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:45.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:45.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:45.887+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:46.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:46.114+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.116764+0000) 2024-02-20T09:22:46.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:46.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:46.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:46.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:46.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:46.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:46.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:46.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:46.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:46.388+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.390890+0000) 2024-02-20T09:22:46.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:46.386+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.388707+0000) 2024-02-20T09:22:46.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:46.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:46.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:46.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:46.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:46.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:46.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:46.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:46.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:46.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:46.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:46.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:46.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:46.890+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:47.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:47.116+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.118772+0000) 2024-02-20T09:22:47.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:47.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:47.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:47.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:47.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:47.165+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:47.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:47.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:47.645 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:47.563+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63197 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:47.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:47.559+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.561882+0000) 2024-02-20T09:22:47.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:47.390+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.392643+0000) 2024-02-20T09:22:47.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:47.560+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.562153+0000) 2024-02-20T09:22:47.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:47.388+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.390578+0000) 2024-02-20T09:22:47.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:47.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:47.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:47.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:47.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:47.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:47.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:47.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:47.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:47.559+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.561663+0000) 2024-02-20T09:22:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:47.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:47.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:47.891+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:47.892+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:48.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:48.117+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:48.118977+0000) 2024-02-20T09:22:48.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:48.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:48.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:48.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:48.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:48.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:48.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:48.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:48.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:48.390+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:48.392839+0000) 2024-02-20T09:22:48.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:48.388+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:48.390792+0000) 2024-02-20T09:22:48.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:48.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:48.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:48.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:48.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:48.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:48.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:48.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:48.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:48.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:48.933+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:48.934+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:49.120+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:49.121440+0000) 2024-02-20T09:22:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:49.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:49.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:49.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:49.170+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:49.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:49.394+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:49.394956+0000) 2024-02-20T09:22:49.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:49.392+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:49.393132+0000) 2024-02-20T09:22:49.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:49.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:49.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:49.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:49.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:49.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:49.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:49.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:50.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:49.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:50.325 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:49.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:50.326 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:49.979+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:50.326 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:49.980+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:50.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:50.122+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:50.123874+0000) 2024-02-20T09:22:50.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:50.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:50.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:50.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:50.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:50.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:50.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:50.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:50.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:50.396+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:50.397398+0000) 2024-02-20T09:22:50.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:50.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:50.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:50.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:50.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:50.365+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:50.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:50.366+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:50.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:50.395+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:50.395933+0000) 2024-02-20T09:22:51.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:50.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:51.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:50.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:51.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:50.935+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:51.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:50.936+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:51.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:51.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:51.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:51.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:51.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:51.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:51.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:51.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:51.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:51.125+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.126475+0000) 2024-02-20T09:22:51.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:51.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:51.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:51.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:51.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:51.397+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.398386+0000) 2024-02-20T09:22:51.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:51.399+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.400206+0000) 2024-02-20T09:22:52.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:51.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:52.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:51.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:52.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:51.965+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:52.118 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:51.966+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:52.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:52.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:52.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:52.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:52.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:52.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:52.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:52.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:52.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:52.127+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.128432+0000) 2024-02-20T09:22:52.693 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:52.572+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63237 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:52.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:52.401+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.402283+0000) 2024-02-20T09:22:52.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:52.569+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.570658+0000) 2024-02-20T09:22:52.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:52.572+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.573027+0000) 2024-02-20T09:22:52.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:52.399+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.400381+0000) 2024-02-20T09:22:52.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:52.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:52.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:52.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:52.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:52.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:52.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:52.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:52.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:52.570+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.571305+0000) 2024-02-20T09:22:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:52.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:52.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:52.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:52.923+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:53.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:53.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:53.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:53.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:53.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:53.102+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:53.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:53.103+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:53.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:53.127+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:53.128638+0000) 2024-02-20T09:22:53.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:53.401+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:53.402490+0000) 2024-02-20T09:22:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:53.399+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:53.400481+0000) 2024-02-20T09:22:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:53.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:53.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:53.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:53.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:54.131 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:53.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:54.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:53.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:54.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:53.924+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:54.132 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:53.925+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:54.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:54.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:54.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:54.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:54.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:54.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:54.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:54.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:54.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:54.129+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:54.130858+0000) 2024-02-20T09:22:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:54.403+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:54.404774+0000) 2024-02-20T09:22:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:54.402+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:54.403811+0000) 2024-02-20T09:22:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:54.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:54.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:54.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:54.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:55.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:54.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:55.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:54.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:55.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:54.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:55.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:54.967+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:55.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:55.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:55.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:55.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:55.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:55.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:55.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:55.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:55.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:55.133+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:55.134059+0000) 2024-02-20T09:22:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:55.407+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:55.408281+0000) 2024-02-20T09:22:55.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:55.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:55.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:55.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:55.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:55.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:55.405+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:55.406087+0000) 2024-02-20T09:22:56.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:56.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:56.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.013+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:56.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.014+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:56.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:56.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:56.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:56.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:56.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:56.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:56.138 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:56.047+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:56.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:56.135+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.136582+0000) 2024-02-20T09:22:56.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:56.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:56.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:56.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:56.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:56.358+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:56.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:56.359+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:56.407+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.408858+0000) 2024-02-20T09:22:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.409+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.410466+0000) 2024-02-20T09:22:57.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:57.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:57.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.980+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:57.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:56.981+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:57.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:57.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:57.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:57.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:57.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:57.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:57.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:57.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:57.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:57.137+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.138917+0000) 2024-02-20T09:22:57.646 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:22:57.580+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63276 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:22:57.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:57.411+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.412441+0000) 2024-02-20T09:22:57.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:57.577+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.578348+0000) 2024-02-20T09:22:57.647 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:57.576+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.577077+0000) 2024-02-20T09:22:57.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:57.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:57.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:57.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:57.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:57.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:57.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:57.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:57.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:57.409+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.410753+0000) 2024-02-20T09:22:57.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:57.577+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.578723+0000) 2024-02-20T09:22:58.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:57.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:57.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:57.943+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:57.943+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:58.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:58.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:58.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:58.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:58.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:58.138+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:58.139131+0000) 2024-02-20T09:22:58.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:58.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:58.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:58.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:58.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:58.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:58.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:58.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:58.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:58.410+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:58.410956+0000) 2024-02-20T09:22:58.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:58.411+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:58.412613+0000) 2024-02-20T09:22:59.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:58.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:58.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:58.966+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:58.967+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:59.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:59.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:59.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:59.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:59.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:22:59.140+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:59.141578+0000) 2024-02-20T09:22:59.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:59.414+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:59.415465+0000) 2024-02-20T09:22:59.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:59.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:59.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:59.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:59.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:59.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:22:59.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:59.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:22:59.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:22:59.412+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:59.413575+0000) 2024-02-20T09:23:00.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:59.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:59.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:59.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:22:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:22:59.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:00.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:00.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:00.026+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:00.027+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:00.142+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:00.143630+0000) 2024-02-20T09:23:00.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:00.416+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:00.417708+0000) 2024-02-20T09:23:00.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:00.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:00.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:00.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:00.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:00.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:00.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:00.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:00.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:00.415+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:00.416366+0000) 2024-02-20T09:23:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:00.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:00.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:00.913+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:00.914+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:01.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:01.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:01.051+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:01.052+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:01.145+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.146104+0000) 2024-02-20T09:23:01.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:01.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:01.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:01.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:01.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:01.374+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:01.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:01.375+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:01.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:01.418+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.419373+0000) 2024-02-20T09:23:01.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:01.420+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.420971+0000) 2024-02-20T09:23:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:01.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:01.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:01.934+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:01.935+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:02.085+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:02.085+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:02.085+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:02.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:02.148+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.149389+0000) 2024-02-20T09:23:02.583 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:02.422+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.423188+0000) 2024-02-20T09:23:02.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:02.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:02.583 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:02.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:02.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:02.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:02.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:02.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:02.584 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:02.420+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.421892+0000) 2024-02-20T09:23:02.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:02.585+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.586196+0000) 2024-02-20T09:23:02.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:02.584+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.585747+0000) 2024-02-20T09:23:02.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:02.584+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.585777+0000) 2024-02-20T09:23:02.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:02.587+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63317 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:03.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:02.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:03.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:02.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:03.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:02.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:03.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:02.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:03.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:03.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:03.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:03.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:03.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:03.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:03.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:03.044+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:03.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:03.148+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:03.149609+0000) 2024-02-20T09:23:03.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:03.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:03.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:03.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:03.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:03.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:03.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:03.348+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:03.421+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:03.422095+0000) 2024-02-20T09:23:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:03.422+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:03.423349+0000) 2024-02-20T09:23:04.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:03.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:04.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:03.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:04.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:03.903+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:04.154 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:03.904+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:04.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:04.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:04.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:04.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:04.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:04.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:04.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:04.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:04.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:04.151+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:04.152415+0000) 2024-02-20T09:23:04.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:04.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:04.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:04.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:04.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:04.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:04.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:04.349+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:04.424+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:04.425027+0000) 2024-02-20T09:23:04.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:04.425+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:04.426132+0000) 2024-02-20T09:23:05.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:04.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:05.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:04.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:05.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:04.894+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:05.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:04.895+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:05.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:05.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:05.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:05.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:05.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:05.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:05.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:05.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:05.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:05.154+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:05.155884+0000) 2024-02-20T09:23:05.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:05.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:05.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:05.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:05.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:05.342+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:05.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:05.343+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:05.427+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:05.428477+0000) 2024-02-20T09:23:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:05.428+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:05.429190+0000) 2024-02-20T09:23:06.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:05.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:06.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:05.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:06.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:05.849+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:06.117 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:05.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:06.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:06.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:06.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:06.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:06.114+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:06.115+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:06.158+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.159123+0000) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:06.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:06.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:06.326+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:06.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:06.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:06.430+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.431209+0000) 2024-02-20T09:23:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:06.431+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.432268+0000) 2024-02-20T09:23:07.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:06.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:07.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:06.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:07.162 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:06.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:07.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:06.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:07.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:07.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:07.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:07.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:07.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:07.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:07.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:07.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:07.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:07.160+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.161310+0000) 2024-02-20T09:23:07.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:07.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:07.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:07.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:07.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:07.364+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:07.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:07.365+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:07.717 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:07.595+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63356 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:07.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:07.432+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.433395+0000) 2024-02-20T09:23:07.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:07.592+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.593684+0000) 2024-02-20T09:23:07.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:07.433+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.434576+0000) 2024-02-20T09:23:07.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:07.592+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.593471+0000) 2024-02-20T09:23:07.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:07.593+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.594098+0000) 2024-02-20T09:23:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:07.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:07.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:07.926+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:07.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:08.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:08.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:08.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:08.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:08.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:08.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:08.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:08.094+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:08.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:08.160+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:08.161510+0000) 2024-02-20T09:23:08.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:08.433+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:08.434766+0000) 2024-02-20T09:23:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:08.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:08.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:08.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:08.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:08.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:08.432+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:08.433589+0000) 2024-02-20T09:23:09.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:08.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:08.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:08.972+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:08.973+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:09.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:09.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:09.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:09.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:09.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:09.427 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:09.162+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:09.163605+0000) 2024-02-20T09:23:09.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:09.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:09.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:09.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:09.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:09.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:09.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:09.389+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:09.435+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:09.436897+0000) 2024-02-20T09:23:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:09.437+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:09.437932+0000) 2024-02-20T09:23:10.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:09.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:10.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:09.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:10.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:09.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:10.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:09.976+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:10.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:10.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:10.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:10.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:10.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:10.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:10.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:10.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:10.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:10.166+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:10.166968+0000) 2024-02-20T09:23:10.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:10.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:10.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:10.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:10.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:10.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:10.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:10.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:10.438+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:10.439900+0000) 2024-02-20T09:23:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:10.440+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:10.441417+0000) 2024-02-20T09:23:11.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:10.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:11.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:10.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:11.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:10.974+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:11.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:10.974+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:11.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:11.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:11.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:11.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:11.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:11.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:11.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:11.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:11.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:11.168+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.169301+0000) 2024-02-20T09:23:11.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:11.349+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:11.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:11.349+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:11.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:11.349+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:11.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:11.350+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:11.441+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.442540+0000) 2024-02-20T09:23:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:11.444+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.445099+0000) 2024-02-20T09:23:12.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:12.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:12.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:12.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:12.111+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:12.112+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:12.170+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.171560+0000) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:11.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:11.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:11.994+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:12.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:11.995+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:12.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:12.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:12.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:12.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:12.427 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:12.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:12.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:12.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:12.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:12.443+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.444657+0000) 2024-02-20T09:23:12.703 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:12.600+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.601822+0000) 2024-02-20T09:23:12.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:12.446+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.447237+0000) 2024-02-20T09:23:12.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:12.600+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.601668+0000) 2024-02-20T09:23:12.704 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:12.603+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63395 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:12.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:12.600+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.601755+0000) 2024-02-20T09:23:13.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:12.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:13.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:12.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:13.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:12.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:13.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:12.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:13.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:13.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:13.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:13.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:13.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:13.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:13.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:13.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:13.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:13.170+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:13.171787+0000) 2024-02-20T09:23:13.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:13.446+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:13.447442+0000) 2024-02-20T09:23:13.648 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:13.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:13.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:13.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:13.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:13.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:13.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:13.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:13.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:13.443+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:13.444874+0000) 2024-02-20T09:23:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:13.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:13.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:13.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:13.949+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:14.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:14.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:14.133+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:14.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:14.174+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:14.175407+0000) 2024-02-20T09:23:14.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:14.449+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:14.449958+0000) 2024-02-20T09:23:14.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:14.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:14.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:14.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:14.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:14.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:14.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:14.387+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:14.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:14.447+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:14.448393+0000) 2024-02-20T09:23:15.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:14.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:15.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:14.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:15.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:14.922+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:15.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:14.922+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:15.180 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:15.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:15.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:15.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:15.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:15.122+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:15.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:15.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:15.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:15.178+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:15.178971+0000) 2024-02-20T09:23:15.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:15.344+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:15.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:15.344+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:15.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:15.344+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:15.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:15.345+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:15.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:15.450+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:15.451295+0000) 2024-02-20T09:23:15.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:15.451+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:15.452668+0000) 2024-02-20T09:23:16.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:15.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:16.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:15.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:16.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:15.948+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:16.104 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:15.949+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:16.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:16.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:16.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:16.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:16.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:16.181+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.182583+0000) 2024-02-20T09:23:16.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:16.454+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.455345+0000) 2024-02-20T09:23:16.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:16.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:16.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:16.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:16.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:16.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:16.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:16.369+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:16.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:16.453+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.454809+0000) 2024-02-20T09:23:17.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:16.988+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:17.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:16.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:17.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:16.989+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:17.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:16.990+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:17.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:17.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:17.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:17.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:17.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:17.184+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.185677+0000) 2024-02-20T09:23:17.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:17.457+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.458340+0000) 2024-02-20T09:23:17.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:17.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:17.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:17.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:17.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:17.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:17.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:17.418+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:17.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:17.456+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.457892+0000) 2024-02-20T09:23:17.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:17.610+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63435 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:17.870 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:17.608+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.609877+0000) 2024-02-20T09:23:17.871 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:17.607+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.608601+0000) 2024-02-20T09:23:17.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:17.609+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.610236+0000) 2024-02-20T09:23:18.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:18.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:18.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:18.038+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:18.039+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:18.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:18.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:18.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:18.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:18.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:18.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:18.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:18.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:18.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:18.184+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:18.185884+0000) 2024-02-20T09:23:18.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:18.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:18.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:18.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:18.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:18.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:18.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:18.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:18.457+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:18.458088+0000) 2024-02-20T09:23:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:18.457+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:18.458566+0000) 2024-02-20T09:23:19.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:19.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:19.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:19.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:19.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:19.069+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:19.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:19.070+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:19.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:19.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:19.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:19.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:19.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:19.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:19.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:19.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:19.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:19.187+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:19.188537+0000) 2024-02-20T09:23:19.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:19.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:19.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:19.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:19.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:19.244+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:19.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:19.244+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:19.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:19.460+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:19.461283+0000) 2024-02-20T09:23:19.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:19.460+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:19.461280+0000) 2024-02-20T09:23:20.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:20.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:20.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:20.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:20.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:20.118+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:20.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:20.119+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:20.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:20.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:20.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:20.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:20.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:20.387+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:20.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:20.388+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:20.190+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:20.191172+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:20.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:20.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:20.204+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:20.205+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:20.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:20.462+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:20.463758+0000) 2024-02-20T09:23:20.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:20.463+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:20.464647+0000) 2024-02-20T09:23:21.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:21.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:21.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:21.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:21.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:21.102+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:21.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:21.103+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:21.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:21.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:21.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:21.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:21.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:21.376+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:21.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:21.377+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:21.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:21.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:21.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:21.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:21.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:21.166+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:21.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:21.166+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:21.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:21.193+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.194052+0000) 2024-02-20T09:23:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:21.466+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.467219+0000) 2024-02-20T09:23:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:21.467+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.468069+0000) 2024-02-20T09:23:22.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:22.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:22.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:22.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:22.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:22.141+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:22.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:22.142+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:22.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:22.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:22.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:22.348+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:22.195+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.196013+0000) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:22.206+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:22.206+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:22.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:22.206+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:22.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:22.207+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:22.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:22.618+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63474 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:22.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:22.615+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.616692+0000) 2024-02-20T09:23:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:22.468+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.469418+0000) 2024-02-20T09:23:22.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:22.615+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.616503+0000) 2024-02-20T09:23:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:22.469+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.470681+0000) 2024-02-20T09:23:22.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:22.616+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.617776+0000) 2024-02-20T09:23:23.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:23.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:23.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:23.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:23.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:23.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:23.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:23.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:23.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:23.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:23.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:23.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:23.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:23.148+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:23.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:23.149+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:23.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:23.195+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:23.196110+0000) 2024-02-20T09:23:23.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:23.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:23.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:23.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:23.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:23.215+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:23.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:23.216+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:23.468+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:23.469629+0000) 2024-02-20T09:23:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:23.469+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:23.470873+0000) 2024-02-20T09:23:24.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:24.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:24.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:24.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:24.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:24.130+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:24.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:24.131+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:24.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:24.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:24.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:24.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:24.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:24.181+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:24.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:24.182+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:24.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:24.198+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:24.199457+0000) 2024-02-20T09:23:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:24.472+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:24.473642+0000) 2024-02-20T09:23:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:24.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:24.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:24.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:24.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:24.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:24.471+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:24.472172+0000) 2024-02-20T09:23:25.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:25.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:25.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:25.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:25.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:25.115+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:25.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:25.116+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:25.183+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:25.183+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:25.183+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:25.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:25.184+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:25.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:25.201+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:25.202735+0000) 2024-02-20T09:23:25.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:25.475+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:25.476480+0000) 2024-02-20T09:23:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:25.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:25.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:25.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:25.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:25.474+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:25.475173+0000) 2024-02-20T09:23:26.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:26.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:26.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:26.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:26.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:26.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:26.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:26.132+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:26.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:26.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:26.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:26.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:26.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:26.188+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:26.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:26.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:26.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:26.204+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.204941+0000) 2024-02-20T09:23:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:26.478+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.479310+0000) 2024-02-20T09:23:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:26.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:26.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:26.453+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:26.454+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:26.476+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.477912+0000) 2024-02-20T09:23:27.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:27.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:27.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:27.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:27.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:27.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:27.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:27.124+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:27.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:27.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:27.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:27.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:27.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:27.195+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:27.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:27.196+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:27.461 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:27.207+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.208047+0000) 2024-02-20T09:23:27.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:27.625+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63514 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:27.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:27.622+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.623298+0000) 2024-02-20T09:23:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:27.480+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.481472+0000) 2024-02-20T09:23:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:27.621+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.622735+0000) 2024-02-20T09:23:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:27.479+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.479974+0000) 2024-02-20T09:23:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:27.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:27.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:27.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:27.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:27.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:27.624+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.625343+0000) 2024-02-20T09:23:28.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:28.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:28.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:28.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:28.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:28.120+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:28.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:28.121+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:28.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:28.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:28.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:28.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:28.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:28.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:28.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:28.173+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:28.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:28.207+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:28.208224+0000) 2024-02-20T09:23:28.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:28.480+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:28.481661+0000) 2024-02-20T09:23:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:28.479+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:28.480078+0000) 2024-02-20T09:23:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:28.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:28.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:28.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:28.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:28.546+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:29.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:29.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:29.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:29.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:29.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:29.147+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:29.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:29.148+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:29.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:29.203+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:29.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:29.203+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:29.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:29.203+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:29.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:29.204+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:29.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:29.209+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:29.210229+0000) 2024-02-20T09:23:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:29.483+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:29.484321+0000) 2024-02-20T09:23:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:29.481+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:29.482464+0000) 2024-02-20T09:23:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:29.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:29.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:29.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:29.585+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:30.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:30.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:30.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:30.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:30.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:30.112+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:30.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:30.113+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:30.211+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:30.212351+0000) 2024-02-20T09:23:30.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:30.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:30.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:30.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:30.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:30.233+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:30.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:30.486+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:30.487512+0000) 2024-02-20T09:23:30.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:30.484+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:30.485049+0000) 2024-02-20T09:23:30.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:30.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:30.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:30.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:30.883 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:30.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:30.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:30.616+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:31.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:31.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:31.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:31.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:31.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:31.135+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:31.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:31.136+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:31.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:31.213+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.214862+0000) 2024-02-20T09:23:31.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:31.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:31.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:31.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:31.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:31.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:31.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:31.219+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:31.489+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.490789+0000) 2024-02-20T09:23:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:31.487+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.488441+0000) 2024-02-20T09:23:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:31.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:31.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:31.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:31.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:32.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:32.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:32.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:32.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:32.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:32.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:32.217+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.217932+0000) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:32.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:32.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:32.243+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:32.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:32.243+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:32.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:32.631+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.632559+0000) 2024-02-20T09:23:32.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:32.633+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63555 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:32.492+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.493914+0000) 2024-02-20T09:23:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:32.629+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.630506+0000) 2024-02-20T09:23:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:32.489+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.490275+0000) 2024-02-20T09:23:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:32.633+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.634179+0000) 2024-02-20T09:23:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:32.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:32.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:32.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:32.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:32.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:33.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:33.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:33.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:33.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:33.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:33.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:33.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:33.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:33.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:33.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:33.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:33.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:33.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:33.212+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:33.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:33.213+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:33.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:33.217+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:33.218106+0000) 2024-02-20T09:23:33.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:33.493+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:33.494108+0000) 2024-02-20T09:23:33.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:33.489+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:33.490450+0000) 2024-02-20T09:23:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:33.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:33.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:33.668+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:33.668+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:34.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:34.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:34.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:34.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:34.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:34.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:34.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:34.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:34.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:34.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:34.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:34.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:34.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:34.169+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:34.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:34.170+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:34.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:34.219+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:34.220544+0000) 2024-02-20T09:23:34.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:34.491+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:34.492583+0000) 2024-02-20T09:23:34.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:34.495+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:34.496220+0000) 2024-02-20T09:23:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:34.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:34.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:34.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:34.700+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:35.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:35.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:35.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:35.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:35.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:35.179+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:35.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:35.180+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:35.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:35.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:35.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:35.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:35.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:35.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:35.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:35.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:35.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:35.221+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:35.222720+0000) 2024-02-20T09:23:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:35.497+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:35.498246+0000) 2024-02-20T09:23:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:35.493+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:35.494709+0000) 2024-02-20T09:23:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:35.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:35.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:35.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:35.673+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:36.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:36.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:36.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:36.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:36.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:36.198+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:36.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:36.199+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:36.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:36.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:36.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:36.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:36.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:36.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:36.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:36.157+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:36.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:36.223+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.224926+0000) 2024-02-20T09:23:36.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:36.496+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.497347+0000) 2024-02-20T09:23:36.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:36.499+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.500754+0000) 2024-02-20T09:23:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:36.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:36.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:36.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:36.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:37.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:37.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:37.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:37.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:37.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:37.191+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:37.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:37.192+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:37.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:37.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:37.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:37.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:37.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:37.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:37.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:37.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:37.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:37.226+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.227345+0000) 2024-02-20T09:23:37.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:37.641+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63595 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:37.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:37.638+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.639054+0000) 2024-02-20T09:23:37.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:37.498+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.499559+0000) 2024-02-20T09:23:37.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:37.637+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.638377+0000) 2024-02-20T09:23:37.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:37.501+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.502707+0000) 2024-02-20T09:23:37.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:37.637+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.638510+0000) 2024-02-20T09:23:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:37.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:37.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:37.719+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:37.720+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:38.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:38.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:38.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:38.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:38.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:38.174+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:38.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:38.175+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:38.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:38.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:38.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:38.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:38.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:38.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:38.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:38.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:38.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:38.226+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:38.227572+0000) 2024-02-20T09:23:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:38.498+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:38.499756+0000) 2024-02-20T09:23:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:38.501+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:38.502898+0000) 2024-02-20T09:23:39.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:38.760+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:39.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:38.760+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:39.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:38.760+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:39.105 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:38.761+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:39.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:39.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:39.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:39.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:39.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:39.154+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:39.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:39.155+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:39.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:39.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:39.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:39.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:39.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:39.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:39.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:39.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:39.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:39.228+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:39.229966+0000) 2024-02-20T09:23:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:39.501+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:39.503185+0000) 2024-02-20T09:23:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:39.503+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:39.505336+0000) 2024-02-20T09:23:40.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:39.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:40.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:39.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:40.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:39.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:40.056 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:39.755+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:40.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:40.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:40.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:40.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:40.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:40.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:40.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:40.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:40.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:40.230+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:40.232076+0000) 2024-02-20T09:23:40.331 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:40.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:40.331 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:40.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:40.331 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:40.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:40.331 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:40.183+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:40.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:40.503+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:40.505807+0000) 2024-02-20T09:23:40.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:40.505+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:40.507819+0000) 2024-02-20T09:23:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:40.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:40.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:40.765+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:40.765+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:41.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:41.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:41.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:41.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:41.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:41.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:41.331 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:41.218+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:41.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:41.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:41.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:41.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:41.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:41.049+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:41.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:41.049+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:41.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:41.233+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.235160+0000) 2024-02-20T09:23:41.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:41.506+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.508460+0000) 2024-02-20T09:23:41.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:41.508+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.510307+0000) 2024-02-20T09:23:41.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:41.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:41.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:41.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:41.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:41.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:41.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:41.756+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:42.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.006+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.007+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:42.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:42.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:42.185+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:42.186+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:42.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.236+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.237357+0000) 2024-02-20T09:23:42.771 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:42.649+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63635 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:42.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.649+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.650016+0000) 2024-02-20T09:23:42.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:42.511+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.512065+0000) 2024-02-20T09:23:42.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:42.646+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.647521+0000) 2024-02-20T09:23:42.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:42.509+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.510357+0000) 2024-02-20T09:23:42.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:42.645+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.646132+0000) 2024-02-20T09:23:42.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:42.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:42.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:42.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:42.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:42.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:42.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:42.723+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:43.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:43.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:43.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.990+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:43.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:42.991+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:43.423 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:43.236+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:43.237561+0000) 2024-02-20T09:23:43.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:43.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:43.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:43.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:43.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:43.165+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:43.424 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:43.166+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:43.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:43.509+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:43.510564+0000) 2024-02-20T09:23:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:43.511+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:43.512245+0000) 2024-02-20T09:23:44.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:43.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:44.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:43.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:44.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:43.987+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:44.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:43.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:44.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:43.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:44.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:43.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:44.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:43.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:44.212 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:43.747+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:44.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:44.239+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:44.240061+0000) 2024-02-20T09:23:44.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:44.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:44.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:44.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:44.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:44.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:44.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:44.210+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:44.512+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:44.513172+0000) 2024-02-20T09:23:44.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:44.513+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:44.514736+0000) 2024-02-20T09:23:45.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:44.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:45.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:44.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:45.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:44.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:45.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:44.771+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:45.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:44.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:45.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:44.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:45.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:44.981+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:45.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:44.982+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:45.462 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:45.241+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:45.242500+0000) 2024-02-20T09:23:45.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:45.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:45.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:45.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:45.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:45.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:45.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:45.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:45.515+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:45.516056+0000) 2024-02-20T09:23:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:45.515+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:45.516914+0000) 2024-02-20T09:23:46.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:45.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:46.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:45.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:46.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:45.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:46.187 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:45.732+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:46.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:45.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:46.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:45.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:46.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:45.976+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:46.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:45.977+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:46.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:46.244+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.245695+0000) 2024-02-20T09:23:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:46.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:46.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:46.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:46.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:46.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:46.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:46.518+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.519605+0000) 2024-02-20T09:23:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:46.517+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.518632+0000) 2024-02-20T09:23:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:46.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:46.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:46.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:46.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:46.699+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:47.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:47.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:47.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:47.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:47.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:47.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:47.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:47.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:47.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:47.247+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.247953+0000) 2024-02-20T09:23:47.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:47.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:47.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:47.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:47.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:47.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:47.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:47.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:47.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:47.656+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63675 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:47.653+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.653968+0000) 2024-02-20T09:23:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:47.521+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.522649+0000) 2024-02-20T09:23:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:47.653+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.654374+0000) 2024-02-20T09:23:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:47.520+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.521227+0000) 2024-02-20T09:23:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:47.654+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.655416+0000) 2024-02-20T09:23:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:47.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:47.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:47.656+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:47.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:47.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:48.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:48.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:48.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:48.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:48.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:48.024+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:48.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:48.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:48.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:48.247+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:48.248126+0000) 2024-02-20T09:23:48.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:48.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:48.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:48.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:48.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:48.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:48.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:48.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:48.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:48.521+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:48.522827+0000) 2024-02-20T09:23:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:48.520+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:48.521445+0000) 2024-02-20T09:23:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:48.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:48.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:48.608+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:48.609+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:49.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:49.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:49.131+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:49.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:49.131+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:49.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:49.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:49.074+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:49.075+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:49.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:49.249+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:49.250518+0000) 2024-02-20T09:23:49.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:49.524+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:49.524958+0000) 2024-02-20T09:23:49.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:49.522+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:49.523741+0000) 2024-02-20T09:23:49.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:49.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:49.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:49.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:49.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:49.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:49.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:49.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:50.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:50.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:50.119+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:50.120+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:50.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:50.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:50.043+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:50.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:50.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:50.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:50.251+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:50.252751+0000) 2024-02-20T09:23:50.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:50.527+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:50.528523+0000) 2024-02-20T09:23:50.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:50.525+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:50.526669+0000) 2024-02-20T09:23:50.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:50.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:50.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:50.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:50.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:50.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:50.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:50.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:51.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:51.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:51.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:51.035+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:51.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:51.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:51.144+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:51.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:51.145+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:51.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:51.254+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.254977+0000) 2024-02-20T09:23:51.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:51.530+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.531659+0000) 2024-02-20T09:23:51.901 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:51.528+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.529359+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:51.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:51.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:51.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:51.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:52.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:52.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:52.110+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:52.111+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:52.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:52.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:52.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:52.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:52.532 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:52.256+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.257367+0000) 2024-02-20T09:23:52.784 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:52.664+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63714 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:52.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:52.532+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.533842+0000) 2024-02-20T09:23:52.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:52.660+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.661035+0000) 2024-02-20T09:23:52.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:52.661+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.662346+0000) 2024-02-20T09:23:52.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:52.530+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.531467+0000) 2024-02-20T09:23:52.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:52.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:52.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:52.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:52.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:52.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:52.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:52.654+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:52.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:52.661+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.662471+0000) 2024-02-20T09:23:53.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:53.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:53.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:53.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:53.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:53.124+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:53.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:53.125+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:53.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:53.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:53.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:53.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:53.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:53.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:53.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:53.086+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:53.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:53.256+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:53.257577+0000) 2024-02-20T09:23:53.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:53.530+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:53.531681+0000) 2024-02-20T09:23:53.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:53.533+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:53.534055+0000) 2024-02-20T09:23:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:53.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:53.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:53.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:53.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:54.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:54.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:54.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:54.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:54.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:54.150+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:54.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:54.150+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:54.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:54.131+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:54.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:54.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:54.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:54.132+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:54.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:54.132+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:54.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:54.258+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:54.259755+0000) 2024-02-20T09:23:54.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:54.536+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:54.537403+0000) 2024-02-20T09:23:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:54.532+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:54.533689+0000) 2024-02-20T09:23:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:54.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:54.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:54.633+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:54.634+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:55.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:55.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:55.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:55.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:55.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:55.167+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:55.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:55.168+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:55.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:55.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:55.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:55.171+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:55.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:55.172+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:55.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:55.261+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:55.262332+0000) 2024-02-20T09:23:55.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:55.538+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:55.539504+0000) 2024-02-20T09:23:55.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:55.536+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:55.537059+0000) 2024-02-20T09:23:55.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:55.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:55.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:55.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:55.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:55.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:55.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:55.641+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:56.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:56.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:56.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:56.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:56.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:56.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:56.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:56.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:56.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:56.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:56.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:56.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:56.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:56.127+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:56.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:56.128+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:56.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:56.265+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.265948+0000) 2024-02-20T09:23:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:56.540+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.541888+0000) 2024-02-20T09:23:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:56.539+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.540214+0000) 2024-02-20T09:23:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:56.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:56.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:56.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:56.629+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:56.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:56.629+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:57.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:57.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:57.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:57.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:57.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:57.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:57.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:57.114+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:57.432 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:57.267+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.268237+0000) 2024-02-20T09:23:57.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:57.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:57.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:57.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:57.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:57.169+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:57.433 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:57.170+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:57.694 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:23:57.671+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63754 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:23:57.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:57.544+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.545138+0000) 2024-02-20T09:23:57.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:57.670+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.671349+0000) 2024-02-20T09:23:57.695 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:57.668+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.669624+0000) 2024-02-20T09:23:57.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:57.541+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.542143+0000) 2024-02-20T09:23:57.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:57.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:57.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:57.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:57.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:57.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:57.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:57.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:57.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:57.669+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.670757+0000) 2024-02-20T09:23:58.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:58.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:58.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:58.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:58.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:58.209+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:58.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:58.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:58.117+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:58.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:58.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:58.267+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:58.268490+0000) 2024-02-20T09:23:58.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:58.544+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:58.545307+0000) 2024-02-20T09:23:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:58.541+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:58.542397+0000) 2024-02-20T09:23:58.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:58.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:58.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:58.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:58.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:58.621+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:59.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:59.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:59.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:59.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:59.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:59.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:23:59.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:59.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:23:59.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:23:59.270+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:59.271192+0000) 2024-02-20T09:23:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:59.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:59.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:59.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:24.752663+0000 (oldest deadline 2024-02-20T09:13:44.752663+0000) 2024-02-20T09:23:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:59.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:59.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:23:59.546+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:59.547461+0000) 2024-02-20T09:23:59.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:59.543+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:59.544836+0000) 2024-02-20T09:23:59.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:59.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:59.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:59.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:59.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:59.618+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:23:59.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:23:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:23:59.619+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:00.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:00.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:00.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:00.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:00.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:00.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:00.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:00.273+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:00.274843+0000) 2024-02-20T09:24:00.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:00.546+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:00.547400+0000) 2024-02-20T09:24:00.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:00.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:00.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:00.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:00.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:00.913 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:00.549+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:00.549966+0000) 2024-02-20T09:24:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:01.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:01.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:01.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:01.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:01.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:01.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:01.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:01.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:01.276+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.277377+0000) 2024-02-20T09:24:01.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:01.551+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.552482+0000) 2024-02-20T09:24:01.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:01.549+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.550050+0000) 2024-02-20T09:24:01.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:01.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:01.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:01.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:01.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:01.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:01.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:01.649+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:02.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:02.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:02.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:02.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:02.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:02.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:02.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:02.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:02.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:02.121+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:02.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:02.279+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.280693+0000) 2024-02-20T09:24:02.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:02.554+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.555911+0000) 2024-02-20T09:24:02.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:02.552+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.553299+0000) 2024-02-20T09:24:02.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:02.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:02.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:02.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:02.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:02.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:02.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:02.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:02.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:02.676+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.677098+0000) 2024-02-20T09:24:02.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:02.677+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.678761+0000) 2024-02-20T09:24:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:02.676+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.677255+0000) 2024-02-20T09:24:02.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:02.680+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63793 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:03.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:03.251+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:03.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:03.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:03.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:03.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:03.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:03.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:03.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:03.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:03.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:03.279+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:03.280894+0000) 2024-02-20T09:24:03.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:03.555+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:03.556134+0000) 2024-02-20T09:24:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:03.552+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:03.553506+0000) 2024-02-20T09:24:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:03.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:03.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:03.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:03.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:03.646+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:03.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:03.647+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:04.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:04.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:04.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:04.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:04.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:04.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:04.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:04.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:04.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:04.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:04.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:04.282+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:04.283343+0000) 2024-02-20T09:24:04.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:04.554+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:04.555758+0000) 2024-02-20T09:24:04.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:04.557+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:04.558201+0000) 2024-02-20T09:24:04.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:04.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:04.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:04.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:04.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:04.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:04.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:04.674+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:05.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:05.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:05.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:05.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:05.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:05.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:05.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:05.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:05.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:05.189+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:05.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:05.284+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:05.285611+0000) 2024-02-20T09:24:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:05.559+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:05.560184+0000) 2024-02-20T09:24:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:05.557+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:05.558452+0000) 2024-02-20T09:24:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:05.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:05.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:05.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:29.302009+0000 (oldest deadline 2024-02-20T09:13:49.302009+0000) 2024-02-20T09:24:05.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:05.678+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:06.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:06.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:06.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:06.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:06.146+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:06.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:06.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:06.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:06.286+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.287746+0000) 2024-02-20T09:24:06.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:06.561+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.562496+0000) 2024-02-20T09:24:06.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:06.560+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.561782+0000) 2024-02-20T09:24:06.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:06.721+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:07.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:07.267+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:07.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:07.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:07.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:07.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:07.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:07.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:07.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:07.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:07.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:07.288+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.289653+0000) 2024-02-20T09:24:07.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:07.687+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63833 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:07.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:07.684+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.685312+0000) 2024-02-20T09:24:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:07.563+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.564466+0000) 2024-02-20T09:24:07.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:07.684+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.685482+0000) 2024-02-20T09:24:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:07.564+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.565617+0000) 2024-02-20T09:24:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:07.685+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.686519+0000) 2024-02-20T09:24:08.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:07.730+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:08.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:08.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:08.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:08.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:08.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:08.213+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:08.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:08.238+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:08.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:08.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:08.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:08.288+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:08.289850+0000) 2024-02-20T09:24:08.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:08.564+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:08.565829+0000) 2024-02-20T09:24:08.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:08.563+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:08.564636+0000) 2024-02-20T09:24:08.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:08.716+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:09.569 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:09.252+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:09.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:09.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:09.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:09.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:09.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:09.238+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:13:38.409247+0000 (oldest deadline 2024-02-20T09:13:58.409247+0000) 2024-02-20T09:24:09.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:09.239+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:09.570 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:09.292+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:09.293114+0000) 2024-02-20T09:24:09.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:09.568+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:09.569301+0000) 2024-02-20T09:24:09.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:09.567+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:09.568179+0000) 2024-02-20T09:24:09.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:09.737+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:10.572 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:10.220+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:10.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:10.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:10.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:10.294+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:10.295689+0000) 2024-02-20T09:24:10.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:10.571+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:10.572027+0000) 2024-02-20T09:24:10.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:10.570+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:10.571298+0000) 2024-02-20T09:24:10.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:10.714+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:11.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:11.196+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:11.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:11.297+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.297947+0000) 2024-02-20T09:24:11.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:11.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:11.573+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.574858+0000) 2024-02-20T09:24:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:11.573+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.574314+0000) 2024-02-20T09:24:12.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:11.747+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:12.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:12.218+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:12.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:12.299+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.300352+0000) 2024-02-20T09:24:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:12.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:12.923 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:12.695+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63872 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:12.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:12.576+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.576969+0000) 2024-02-20T09:24:12.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:12.692+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.693042+0000) 2024-02-20T09:24:12.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:12.691+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.692798+0000) 2024-02-20T09:24:12.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:12.576+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.577078+0000) 2024-02-20T09:24:12.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:12.693+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.694333+0000) 2024-02-20T09:24:12.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:12.779+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:13.578 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:13.263+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:13.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:13.299+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:13.300563+0000) 2024-02-20T09:24:13.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:13.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:13.844 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:13.576+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:13.577185+0000) 2024-02-20T09:24:13.844 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:13.576+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:13.577284+0000) 2024-02-20T09:24:13.845 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:13.750+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:14.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:14.260+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:14.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:14.301+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:14.302753+0000) 2024-02-20T09:24:14.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:14.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:14.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:14.578+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:14.579771+0000) 2024-02-20T09:24:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:14.579+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:14.580266+0000) 2024-02-20T09:24:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:14.704+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:15.584 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:15.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:15.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:15.304+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:15.304964+0000) 2024-02-20T09:24:15.584 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:15.333+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:15.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:15.582+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:15.583326+0000) 2024-02-20T09:24:15.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:15.582+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:15.583740+0000) 2024-02-20T09:24:15.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:15.664+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:16.217+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:16.306+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.307490+0000) 2024-02-20T09:24:16.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:16.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:16.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:16.584+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.585513+0000) 2024-02-20T09:24:16.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:16.585+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.586352+0000) 2024-02-20T09:24:16.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:16.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:17.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:17.176+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:17.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:17.308+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.309372+0000) 2024-02-20T09:24:17.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:17.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:17.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:17.586+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.587447+0000) 2024-02-20T09:24:17.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:17.587+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.588774+0000) 2024-02-20T09:24:17.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:17.664+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:17.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:17.703+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63912 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:17.701+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.702685+0000) 2024-02-20T09:24:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:17.702+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.703766+0000) 2024-02-20T09:24:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:17.701+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.702215+0000) 2024-02-20T09:24:18.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:18.135+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:18.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:18.282+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:18.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:18.308+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:18.309585+0000) 2024-02-20T09:24:18.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:18.586+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:18.587657+0000) 2024-02-20T09:24:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:18.588+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:18.588957+0000) 2024-02-20T09:24:18.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:18.651+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:19.146+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:19.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:19.275+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:19.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:19.312+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:19.313128+0000) 2024-02-20T09:24:19.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:19.589+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:19.590714+0000) 2024-02-20T09:24:19.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:19.590+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:19.591387+0000) 2024-02-20T09:24:19.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:19.624+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:20.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:20.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:20.178+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:20.179+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:20.594 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:20.283+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:20.594 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:20.315+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:20.316706+0000) 2024-02-20T09:24:20.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:20.592+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:20.593347+0000) 2024-02-20T09:24:20.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:20.594+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:20.594954+0000) 2024-02-20T09:24:20.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:20.626+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:21.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:21.298+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:21.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:21.318+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.319031+0000) 2024-02-20T09:24:21.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:21.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:21.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:21.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:21.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:21.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:21.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:21.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:21.596+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.596944+0000) 2024-02-20T09:24:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:21.596+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.597425+0000) 2024-02-20T09:24:21.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:21.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:22.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:22.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:22.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:22.320+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.321341+0000) 2024-02-20T09:24:22.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:22.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:22.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:22.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:22.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:22.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:22.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:22.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:22.720 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:22.711+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63951 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:22.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:22.709+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.709932+0000) 2024-02-20T09:24:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:22.598+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.599522+0000) 2024-02-20T09:24:22.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:22.708+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.709546+0000) 2024-02-20T09:24:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:22.598+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.599747+0000) 2024-02-20T09:24:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:22.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:22.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:22.709+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.709934+0000) 2024-02-20T09:24:23.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:23.267+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:23.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:23.320+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:23.321504+0000) 2024-02-20T09:24:23.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:23.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:23.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:23.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:23.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:23.166+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:23.435 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:23.167+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:23.598+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:23.599720+0000) 2024-02-20T09:24:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:23.599+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:23.599932+0000) 2024-02-20T09:24:23.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:23.644+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:24.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:24.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:24.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:24.323+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:24.324842+0000) 2024-02-20T09:24:24.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:24.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:24.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:24.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:24.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:24.184+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:24.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:24.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:24.602+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:24.603126+0000) 2024-02-20T09:24:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:24.602+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:24.603469+0000) 2024-02-20T09:24:24.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:24.605+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:25.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:25.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:25.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:25.145+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:25.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:25.147+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:25.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:25.570+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:25.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:25.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:25.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:25.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:25.327596+0000) 2024-02-20T09:24:25.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:25.605+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:25.606542+0000) 2024-02-20T09:24:25.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:25.604+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:25.605776+0000) 2024-02-20T09:24:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:26.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:26.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:26.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:26.180+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:26.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:26.181+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:26.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:26.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:26.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:26.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:26.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:26.328+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.329754+0000) 2024-02-20T09:24:26.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:26.608+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.609742+0000) 2024-02-20T09:24:26.937 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:26.608+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.609044+0000) 2024-02-20T09:24:27.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:27.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:27.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:27.330+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.331749+0000) 2024-02-20T09:24:27.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:27.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:27.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:27.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:27.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:27.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:27.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:27.209+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:27.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:27.611+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.612305+0000) 2024-02-20T09:24:27.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:27.572+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:27.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:27.611+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.612447+0000) 2024-02-20T09:24:27.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:27.716+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.716944+0000) 2024-02-20T09:24:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:27.716+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.717405+0000) 2024-02-20T09:24:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:27.717+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.718896+0000) 2024-02-20T09:24:27.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:27.719+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 63991 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:28.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:28.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:28.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:28.331+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:28.331947+0000) 2024-02-20T09:24:28.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:28.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:28.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:28.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:28.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:28.229+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:28.552 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:28.230+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:28.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:28.611+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:28.612549+0000) 2024-02-20T09:24:28.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:28.549+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:28.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:28.611+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:28.612677+0000) 2024-02-20T09:24:29.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:29.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:29.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:29.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:29.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:29.202+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:29.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:29.203+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:29.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:29.314+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:29.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:29.333+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:29.334493+0000) 2024-02-20T09:24:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:29.614+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:29.615844+0000) 2024-02-20T09:24:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:29.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:29.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:29.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:29.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:29.614+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:29.615261+0000) 2024-02-20T09:24:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:30.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:30.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:30.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:30.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:30.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:30.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:30.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:30.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:30.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:30.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:30.619 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:30.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:30.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:30.524+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:30.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:30.337+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:30.338033+0000) 2024-02-20T09:24:30.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:30.340+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:30.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:30.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:30.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:30.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:30.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:30.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:30.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:30.617+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:30.618317+0000) 2024-02-20T09:24:30.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:30.618+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:30.619044+0000) 2024-02-20T09:24:31.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:31.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:31.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:31.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:31.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:31.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:31.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:31.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:31.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:31.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:31.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:31.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:31.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:31.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:31.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:31.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:31.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:31.340+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.340954+0000) 2024-02-20T09:24:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:31.620+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.622739+0000) 2024-02-20T09:24:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:31.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:31.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:31.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:31.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:31.620+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.621955+0000) 2024-02-20T09:24:32.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:32.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:32.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:32.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:32.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:32.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:32.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:32.219+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:32.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:32.341+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.343205+0000) 2024-02-20T09:24:32.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:32.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:32.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:32.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:32.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:32.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:32.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:32.357+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:32.848 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:32.623+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.625251+0000) 2024-02-20T09:24:32.848 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:32.722+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.724925+0000) 2024-02-20T09:24:32.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:32.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:32.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:32.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:32.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:32.499+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:32.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:32.500+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:32.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:32.623+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.624940+0000) 2024-02-20T09:24:32.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:32.724+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.726167+0000) 2024-02-20T09:24:32.849 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:32.726+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64030 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:32.850 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:32.722+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.724683+0000) 2024-02-20T09:24:33.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:33.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:33.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:33.206+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:33.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:33.207+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:33.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:33.341+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:33.343295+0000) 2024-02-20T09:24:33.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:33.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:33.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:33.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:33.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:33.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:33.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:33.345+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:33.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:33.623+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:33.625415+0000) 2024-02-20T09:24:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:33.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:33.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:33.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:33.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:33.508+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:33.623+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:33.625101+0000) 2024-02-20T09:24:34.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:34.344+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:34.346666+0000) 2024-02-20T09:24:34.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:34.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:34.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:34.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:34.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:34.365+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:34.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:34.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:34.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:34.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:34.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:34.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:34.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:34.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:34.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:34.625+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:34.627818+0000) 2024-02-20T09:24:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:34.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:34.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:34.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:34.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:34.626+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:34.628634+0000) 2024-02-20T09:24:35.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:35.347+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:35.348809+0000) 2024-02-20T09:24:35.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:35.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:35.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:35.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:35.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:35.368+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:35.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:35.369+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:35.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:35.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:35.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:35.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:35.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:35.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:35.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:35.210+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:35.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:35.629+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:35.630625+0000) 2024-02-20T09:24:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:35.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:35.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:35.473+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:35.474+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:35.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:35.630+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:35.631443+0000) 2024-02-20T09:24:36.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:36.219+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:36.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:36.219+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:36.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:36.219+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:36.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:36.220+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:36.486 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:36.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:36.487 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:36.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:36.487 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:36.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:36.487 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:36.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:36.487 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:36.350+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.351349+0000) 2024-02-20T09:24:36.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:36.483+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:36.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:36.483+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:36.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:36.483+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:36.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:36.484+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:36.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:36.633+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.634582+0000) 2024-02-20T09:24:36.948 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:36.632+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.633348+0000) 2024-02-20T09:24:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:37.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:37.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:37.258+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:37.259+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:37.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:37.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:37.492+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:37.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:37.493+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:37.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:37.352+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.353866+0000) 2024-02-20T09:24:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:37.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:37.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:37.361+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:37.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:37.362+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:37.854 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:37.734+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64070 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:37.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:37.730+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.731651+0000) 2024-02-20T09:24:37.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:37.635+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.636694+0000) 2024-02-20T09:24:37.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:37.731+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.732930+0000) 2024-02-20T09:24:37.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:37.635+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.636159+0000) 2024-02-20T09:24:37.856 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:37.732+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.733303+0000) 2024-02-20T09:24:38.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:38.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:38.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:38.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:38.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:38.541+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:38.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:38.542+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:38.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:38.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:38.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:38.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:38.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:38.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:38.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:38.299+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:38.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:38.353+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:38.353974+0000) 2024-02-20T09:24:38.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:38.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:38.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:38.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:38.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:38.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:38.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:38.384+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:38.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:38.635+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:38.636889+0000) 2024-02-20T09:24:38.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:38.635+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:38.636360+0000) 2024-02-20T09:24:39.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:39.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:39.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:39.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:39.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:39.506+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:39.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:39.507+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:39.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:39.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:39.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:39.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:39.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:39.348+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:39.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:39.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:39.338+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:39.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:39.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:39.356+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:39.357433+0000) 2024-02-20T09:24:39.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:39.638+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:39.639095+0000) 2024-02-20T09:24:39.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:39.639+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:39.639954+0000) 2024-02-20T09:24:40.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:40.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:40.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:40.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:40.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:40.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:40.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:40.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:40.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:40.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:40.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:40.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:40.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:40.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:40.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:40.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:40.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:40.358+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:40.359533+0000) 2024-02-20T09:24:40.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:40.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:40.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:40.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:40.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:40.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:40.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:40.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:40.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:40.641+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:40.642422+0000) 2024-02-20T09:24:40.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:40.641+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:40.641976+0000) 2024-02-20T09:24:41.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:41.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:41.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:41.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:41.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:41.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:41.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:41.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:41.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:41.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:41.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:41.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:41.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:41.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:41.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:41.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:41.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:41.361+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.361941+0000) 2024-02-20T09:24:41.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:41.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:41.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:41.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:41.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:41.364+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:41.648 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:41.365+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:41.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:41.644+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.645696+0000) 2024-02-20T09:24:41.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:41.644+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.645184+0000) 2024-02-20T09:24:42.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:42.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:42.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:42.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:42.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:42.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:42.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:42.480+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:42.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:42.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:42.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:42.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:42.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:42.392+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:42.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:42.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:42.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:42.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:42.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:42.363+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.364392+0000) 2024-02-20T09:24:42.953 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:42.742+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64109 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:42.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:42.739+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.740269+0000) 2024-02-20T09:24:42.953 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:42.647+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.648512+0000) 2024-02-20T09:24:42.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:42.738+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.739643+0000) 2024-02-20T09:24:42.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:42.648+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.649587+0000) 2024-02-20T09:24:42.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:42.739+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.740580+0000) 2024-02-20T09:24:43.649 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:43.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:43.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:43.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:43.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:43.507+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:43.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:43.507+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:43.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:43.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:43.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:43.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:43.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:43.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:43.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:43.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:43.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:43.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:43.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:43.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:43.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:43.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:43.652 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:43.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:43.652 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:43.363+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:43.364598+0000) 2024-02-20T09:24:43.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:43.647+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:43.648801+0000) 2024-02-20T09:24:43.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:43.648+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:43.649780+0000) 2024-02-20T09:24:44.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:44.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:44.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:44.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:44.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:44.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:44.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:44.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:44.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:44.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:44.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:44.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:44.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:44.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:44.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:44.337+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:44.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:44.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:44.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:44.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:44.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:44.329+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:44.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:44.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:44.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:44.367+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:44.368104+0000) 2024-02-20T09:24:44.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:44.650+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:44.651557+0000) 2024-02-20T09:24:44.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:44.651+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:44.652690+0000) 2024-02-20T09:24:45.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:45.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:45.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:45.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:45.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:45.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:45.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:45.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:45.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:45.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:45.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:45.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:45.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:45.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:45.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:45.322+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:45.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:45.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:45.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:45.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:45.657 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:45.317+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:45.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:45.318+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:45.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:45.370+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:45.371705+0000) 2024-02-20T09:24:45.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:45.654+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:45.654932+0000) 2024-02-20T09:24:45.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:45.654+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:45.655225+0000) 2024-02-20T09:24:46.658 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:46.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:46.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:46.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:46.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:46.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:46.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:46.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:46.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:46.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:46.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:46.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:46.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:46.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:46.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:46.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:46.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:46.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:46.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:46.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:46.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:46.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:46.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:46.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:46.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:46.372+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.373896+0000) 2024-02-20T09:24:46.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:46.657+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.658691+0000) 2024-02-20T09:24:46.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:46.657+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.657930+0000) 2024-02-20T09:24:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:47.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:47.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:47.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:47.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:47.428+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:47.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:47.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:47.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:47.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:47.388+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:47.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:47.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:47.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:47.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:47.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:47.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:47.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:47.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:47.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:47.375+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.376384+0000) 2024-02-20T09:24:47.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:47.750+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64149 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:47.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:47.747+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.748331+0000) 2024-02-20T09:24:47.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:47.659+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.660831+0000) 2024-02-20T09:24:47.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:47.746+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.747120+0000) 2024-02-20T09:24:47.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:47.659+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.660711+0000) 2024-02-20T09:24:47.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:47.749+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.750194+0000) 2024-02-20T09:24:48.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:48.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:48.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:48.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:48.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:48.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:48.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:48.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:48.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:48.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:48.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:48.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:48.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:48.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:48.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:48.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:48.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:48.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:48.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:48.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:48.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:48.261+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:48.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:48.261+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:48.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:48.375+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:48.376591+0000) 2024-02-20T09:24:48.960 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:48.660+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:48.661034+0000) 2024-02-20T09:24:48.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:48.659+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:48.660921+0000) 2024-02-20T09:24:49.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:49.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:49.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:49.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:49.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:49.443+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:49.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:49.443+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:49.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:49.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:49.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:49.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:49.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:49.304+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:49.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:49.306+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:49.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:49.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:49.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:49.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:49.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:49.259+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:49.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:49.260+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:49.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:49.377+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:49.378819+0000) 2024-02-20T09:24:49.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:49.662+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:49.663533+0000) 2024-02-20T09:24:49.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:49.662+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:49.663411+0000) 2024-02-20T09:24:50.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:50.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:50.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:50.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:50.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:50.413+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:50.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:50.414+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:50.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:50.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:50.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:50.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:50.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:50.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:50.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:50.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:50.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:50.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:50.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:50.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:50.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:50.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:50.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:50.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:50.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:50.380+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:50.381537+0000) 2024-02-20T09:24:50.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:50.665+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:50.666380+0000) 2024-02-20T09:24:50.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:50.665+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:50.666252+0000) 2024-02-20T09:24:51.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:51.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:51.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:51.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:51.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:51.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:51.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:51.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:51.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:51.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:51.384+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:51.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:51.385+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:51.219+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:51.219+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:51.219+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:51.220+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:51.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:51.383+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.383973+0000) 2024-02-20T09:24:51.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:51.668+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.668967+0000) 2024-02-20T09:24:51.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:51.668+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.669135+0000) 2024-02-20T09:24:52.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:52.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:52.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:52.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:52.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:52.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:52.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:52.418+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:52.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:52.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:52.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:52.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:52.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:52.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:52.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:52.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:52.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:52.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:52.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:52.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:52.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:52.242+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:52.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:52.243+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:52.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:52.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.387035+0000) 2024-02-20T09:24:52.962 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:52.757+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64188 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:52.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:52.670+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.671646+0000) 2024-02-20T09:24:52.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:52.755+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.756079+0000) 2024-02-20T09:24:52.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:52.670+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.671667+0000) 2024-02-20T09:24:52.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:52.753+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.754819+0000) 2024-02-20T09:24:52.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:52.755+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.756405+0000) 2024-02-20T09:24:53.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:53.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:53.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:53.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:53.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:53.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:53.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:53.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:53.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:53.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:53.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:53.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:53.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:53.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:53.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:53.416+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:53.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:53.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:53.281+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:53.282+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:53.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:53.386+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:53.387234+0000) 2024-02-20T09:24:53.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:53.670+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:53.671854+0000) 2024-02-20T09:24:53.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:53.670+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:53.671873+0000) 2024-02-20T09:24:54.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:54.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:54.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:54.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:54.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:54.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:54.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:54.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:54.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:54.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:54.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:54.451+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:54.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:54.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:54.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:54.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:54.303+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:54.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:54.389+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:54.390812+0000) 2024-02-20T09:24:54.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:54.674+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:54.674971+0000) 2024-02-20T09:24:54.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:54.674+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:54.675400+0000) 2024-02-20T09:24:55.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:55.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:55.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:55.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:55.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:55.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:55.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:55.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:55.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:55.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:55.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:55.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:55.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:55.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:55.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:55.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:55.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:55.306+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:55.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:55.392+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:55.393327+0000) 2024-02-20T09:24:55.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:55.677+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:55.678296+0000) 2024-02-20T09:24:55.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:55.676+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:55.677807+0000) 2024-02-20T09:24:56.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:56.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:56.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:56.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:56.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:56.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:56.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:56.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:56.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:56.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:56.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:56.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:56.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:56.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:56.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:56.417+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:56.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:56.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:56.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:56.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:56.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:56.288+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:56.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:56.289+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:56.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:56.395+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.396713+0000) 2024-02-20T09:24:56.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:56.680+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.681235+0000) 2024-02-20T09:24:56.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:56.680+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.681085+0000) 2024-02-20T09:24:57.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:57.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:57.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:57.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:57.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:57.377+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:57.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:57.378+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:57.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:57.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:57.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:57.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:57.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:57.452+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:57.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:57.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:57.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:57.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:57.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:57.398+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.399095+0000) 2024-02-20T09:24:57.967 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:24:57.765+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64228 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:24:57.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:57.763+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.764095+0000) 2024-02-20T09:24:57.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:57.683+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.684136+0000) 2024-02-20T09:24:57.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:57.761+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.762584+0000) 2024-02-20T09:24:57.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:57.683+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.684153+0000) 2024-02-20T09:24:57.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:57.762+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.763165+0000) 2024-02-20T09:24:58.659 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:58.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:58.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:58.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:58.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:58.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:58.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:58.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:58.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:58.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:58.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:58.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:58.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:58.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:58.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:58.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:58.398+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:58.399308+0000) 2024-02-20T09:24:58.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:58.683+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:58.684374+0000) 2024-02-20T09:24:58.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:58.683+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:58.684373+0000) 2024-02-20T09:24:59.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:59.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:59.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:59.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:59.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:59.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:24:59.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:59.386+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:59.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:59.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:59.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:59.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:59.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:59.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:59.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:59.299+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:24:59.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:59.300+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:24:59.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:24:59.401+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:59.401978+0000) 2024-02-20T09:24:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:24:59.686+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:59.687834+0000) 2024-02-20T09:24:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:24:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:24:59.686+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:59.687020+0000) 2024-02-20T09:25:00.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:00.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:00.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:00.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:00.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:00.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:00.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:00.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:00.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:00.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:00.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:00.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:00.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:00.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:00.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:00.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:00.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:00.403+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:00.404768+0000) 2024-02-20T09:25:00.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:00.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:00.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:00.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:00.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:00.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:00.690+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:00.691087+0000) 2024-02-20T09:25:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:00.689+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:00.689972+0000) 2024-02-20T09:25:01.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:01.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:01.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:01.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:01.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:01.446+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:01.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:01.447+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:01.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:01.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:01.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:01.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:01.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:01.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:01.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:01.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:01.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:01.406+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.407399+0000) 2024-02-20T09:25:01.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:01.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:01.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:01.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:01.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:01.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:01.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:01.438+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:01.969 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:01.693+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.694686+0000) 2024-02-20T09:25:01.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:01.691+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.692792+0000) 2024-02-20T09:25:02.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:02.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:02.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:02.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:02.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:02.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:02.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:02.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:02.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:02.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:02.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:02.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:02.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:02.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:02.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:02.412+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:02.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:02.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:02.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:02.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:02.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:02.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:02.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:02.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:02.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:02.409+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.410330+0000) 2024-02-20T09:25:02.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:02.773+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64267 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:02.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:02.770+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.771188+0000) 2024-02-20T09:25:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:02.696+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.697156+0000) 2024-02-20T09:25:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:02.771+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.771954+0000) 2024-02-20T09:25:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:02.694+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.695139+0000) 2024-02-20T09:25:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:02.770+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.771427+0000) 2024-02-20T09:25:03.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:03.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:03.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:03.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:03.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:03.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:03.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:03.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:03.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:03.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:03.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:03.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:03.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:03.518+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:03.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:03.519+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:03.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:03.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:03.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:03.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:03.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:03.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:03.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:03.308+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:03.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:03.409+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:03.410532+0000) 2024-02-20T09:25:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:03.696+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:03.697318+0000) 2024-02-20T09:25:03.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:03.694+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:03.695322+0000) 2024-02-20T09:25:04.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:04.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:04.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:04.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:04.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:04.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:04.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:04.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:04.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:04.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:04.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:04.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:04.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:04.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:04.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:04.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:04.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:04.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:04.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:04.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:04.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:04.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:04.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:04.311+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:04.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:04.412+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:04.413866+0000) 2024-02-20T09:25:04.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:04.699+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:04.700489+0000) 2024-02-20T09:25:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:04.697+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:04.698668+0000) 2024-02-20T09:25:05.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:05.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:05.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:05.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:05.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:05.398+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:05.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:05.399+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:05.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:05.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:05.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:05.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:05.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:05.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:05.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:05.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:05.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:05.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:05.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:05.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:05.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:05.322+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:05.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:05.322+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:05.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:05.414+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:05.415881+0000) 2024-02-20T09:25:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:05.701+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:05.701968+0000) 2024-02-20T09:25:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:05.702+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:05.703502+0000) 2024-02-20T09:25:06.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:06.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:06.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:06.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:06.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:06.354+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:06.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:06.355+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:06.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:06.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:06.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:06.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:06.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:06.400+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:06.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:06.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:06.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:06.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:06.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:06.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:06.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:06.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:06.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:06.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:06.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:06.416+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.417876+0000) 2024-02-20T09:25:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:06.705+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.706592+0000) 2024-02-20T09:25:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:06.704+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.705569+0000) 2024-02-20T09:25:07.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:07.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:07.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:07.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:07.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:07.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:07.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:07.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:07.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:07.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:07.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:07.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:07.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:07.322+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:07.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:07.323+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:07.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:07.354+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:07.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:07.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:07.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:07.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:07.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:07.356+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:07.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:07.419+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.420226+0000) 2024-02-20T09:25:07.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:07.781+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64307 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:07.776+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.777650+0000) 2024-02-20T09:25:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:07.708+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.709327+0000) 2024-02-20T09:25:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:07.777+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.778812+0000) 2024-02-20T09:25:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:07.707+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.708279+0000) 2024-02-20T09:25:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:07.779+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.780055+0000) 2024-02-20T09:25:08.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:08.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:08.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:08.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:08.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:08.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:08.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:08.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:08.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:08.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:08.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:08.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:08.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:08.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:08.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:08.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:08.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:08.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:08.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:08.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:08.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:08.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:08.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:08.399+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:08.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:08.419+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:08.420425+0000) 2024-02-20T09:25:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:08.708+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:08.709502+0000) 2024-02-20T09:25:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:08.707+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:08.708466+0000) 2024-02-20T09:25:09.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:09.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:09.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:09.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:09.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:09.332+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:09.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:09.333+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:09.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:09.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:09.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:09.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:09.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:09.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:09.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:09.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:09.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:09.390+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:09.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:09.422+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:09.423026+0000) 2024-02-20T09:25:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:09.711+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:09.712184+0000) 2024-02-20T09:25:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:09.709+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:09.710889+0000) 2024-02-20T09:25:10.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:10.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:10.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:10.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:10.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:10.381+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:10.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:10.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:10.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:10.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:10.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:10.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:10.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:10.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:10.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:10.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:10.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:10.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:10.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:10.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:10.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:10.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:10.424+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:10.425159+0000) 2024-02-20T09:25:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:10.714+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:10.715075+0000) 2024-02-20T09:25:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:10.713+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:10.714229+0000) 2024-02-20T09:25:11.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:11.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:11.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:11.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:11.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:11.405+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:11.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:11.406+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:11.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:11.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:11.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:11.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:11.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:11.457+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:11.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:11.458+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:11.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:11.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:11.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:11.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:11.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:11.383+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:11.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:11.384+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:11.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:11.426+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.427335+0000) 2024-02-20T09:25:11.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:11.717+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.718101+0000) 2024-02-20T09:25:11.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:11.716+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.717292+0000) 2024-02-20T09:25:12.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:12.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:12.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:12.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:12.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:12.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:12.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:12.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:12.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:12.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:12.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:12.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:12.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:12.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:12.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:12.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:12.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:12.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:12.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:12.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:12.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:12.351+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:12.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:12.352+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:12.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:12.429+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.430822+0000) 2024-02-20T09:25:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:12.788+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64346 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:12.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:12.787+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.787958+0000) 2024-02-20T09:25:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:12.720+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.721292+0000) 2024-02-20T09:25:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:12.786+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.787517+0000) 2024-02-20T09:25:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:12.719+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.720486+0000) 2024-02-20T09:25:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:12.785+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.786879+0000) 2024-02-20T09:25:13.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:13.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:13.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:13.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:13.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:13.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:13.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:13.396+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:13.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:13.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:13.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:13.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:13.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:13.429+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:13.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:13.430+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:13.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:13.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:13.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:13.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:13.393+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:13.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:13.430+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:13.431007+0000) 2024-02-20T09:25:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:13.720+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:13.721487+0000) 2024-02-20T09:25:13.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:13.719+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:13.720686+0000) 2024-02-20T09:25:14.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:14.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:14.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:14.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:14.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:14.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:14.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:14.390+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:14.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:14.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:14.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:14.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:14.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:14.423+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:14.432+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:14.433613+0000) 2024-02-20T09:25:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:14.723+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:14.724478+0000) 2024-02-20T09:25:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:14.722+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:14.723356+0000) 2024-02-20T09:25:15.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:15.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:15.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:15.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:15.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:15.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:15.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:15.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:15.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:15.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:15.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:15.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:15.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:15.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:15.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:15.382+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:15.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:15.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:15.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:15.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:15.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:15.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:15.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:15.382+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:15.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:15.435+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:15.435954+0000) 2024-02-20T09:25:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:15.726+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:15.727658+0000) 2024-02-20T09:25:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:15.725+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:15.726012+0000) 2024-02-20T09:25:16.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:16.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:16.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:16.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:16.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:16.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:16.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:16.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:16.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:16.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:16.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:16.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:16.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:16.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:16.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:16.393+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:16.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:16.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:16.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:16.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:16.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:16.429+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:16.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:16.430+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:16.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:16.438+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.438956+0000) 2024-02-20T09:25:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:16.730+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.731135+0000) 2024-02-20T09:25:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:16.728+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.729008+0000) 2024-02-20T09:25:17.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:17.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:17.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:17.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:17.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:17.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:17.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:17.432+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:17.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:17.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:17.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:17.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:17.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:17.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:17.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:17.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:17.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:17.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:17.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:17.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:17.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:17.392+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:17.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:17.392+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:17.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:17.440+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.440959+0000) 2024-02-20T09:25:17.916 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:17.796+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64386 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:17.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:17.793+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.794132+0000) 2024-02-20T09:25:17.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:17.731+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.732296+0000) 2024-02-20T09:25:17.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:17.793+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.794805+0000) 2024-02-20T09:25:17.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:17.732+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.733656+0000) 2024-02-20T09:25:17.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:17.792+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.793849+0000) 2024-02-20T09:25:18.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:18.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:18.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:18.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:18.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:18.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:18.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:18.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:18.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:18.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:18.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:18.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:18.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:18.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:18.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:18.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:18.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:18.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:18.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:18.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:18.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:18.398+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:18.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:18.399+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:18.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:18.440+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:18.441140+0000) 2024-02-20T09:25:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:18.732+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:18.733834+0000) 2024-02-20T09:25:18.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:18.731+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:18.732489+0000) 2024-02-20T09:25:19.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:19.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:19.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:19.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:19.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:19.392+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:19.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:19.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:19.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:19.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:19.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:19.440+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:19.441+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:19.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:19.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:19.394+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:19.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:19.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:19.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:19.443+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:19.444698+0000) 2024-02-20T09:25:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:19.734+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:19.735119+0000) 2024-02-20T09:25:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:19.736+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:19.737082+0000) 2024-02-20T09:25:20.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:20.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:20.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:20.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:20.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:20.408+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:20.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:20.409+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:20.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:20.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:20.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:20.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:20.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:20.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:20.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:20.392+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:20.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:20.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:20.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:20.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:20.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:20.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:20.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:20.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:20.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:20.446+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:20.447248+0000) 2024-02-20T09:25:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:20.739+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:20.740044+0000) 2024-02-20T09:25:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:20.737+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:20.738481+0000) 2024-02-20T09:25:21.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:21.390+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:21.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:21.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:21.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:21.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:21.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:21.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:21.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:21.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:21.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:21.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:21.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:21.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:21.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:21.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:21.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:21.414+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:21.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:21.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:21.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:21.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:21.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:21.416+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:21.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:21.448+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.449300+0000) 2024-02-20T09:25:21.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:21.742+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.743253+0000) 2024-02-20T09:25:21.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:21.740+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.740971+0000) 2024-02-20T09:25:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:22.437+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:22.437+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:22.437+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:22.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:22.438+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:22.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:22.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:22.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:22.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:22.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:22.333+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:22.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:22.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:22.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:22.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:22.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:22.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:22.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:22.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:22.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:22.412+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:22.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:22.450+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.451368+0000) 2024-02-20T09:25:22.925 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:22.804+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64425 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:22.926 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:22.800+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.801886+0000) 2024-02-20T09:25:22.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:22.745+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.745948+0000) 2024-02-20T09:25:22.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:22.800+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.801280+0000) 2024-02-20T09:25:22.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:22.742+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.743196+0000) 2024-02-20T09:25:22.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:22.802+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.803173+0000) 2024-02-20T09:25:23.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:23.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:23.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:23.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:23.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:23.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:23.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:23.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:23.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:23.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:23.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:23.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:23.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:23.350+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:23.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:23.351+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:23.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:23.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:23.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:23.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:23.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:23.384+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:23.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:23.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:23.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:23.450+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:23.451564+0000) 2024-02-20T09:25:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:23.745+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:23.746123+0000) 2024-02-20T09:25:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:23.742+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:23.743412+0000) 2024-02-20T09:25:24.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:24.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:24.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:24.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:24.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:24.525+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:24.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:24.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:24.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:24.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:24.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:24.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:24.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:24.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:24.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:24.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:24.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:24.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:24.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:24.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:24.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:24.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:24.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:24.431+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:24.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:24.452+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:24.453679+0000) 2024-02-20T09:25:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:24.745+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:24.746957+0000) 2024-02-20T09:25:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:24.746+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:24.748590+0000) 2024-02-20T09:25:25.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:25.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:25.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:25.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:25.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:25.528+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:25.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:25.529+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:25.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:25.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:25.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:25.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:25.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:25.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:25.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:25.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:25.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:25.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:25.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:25.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:25.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:25.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:25.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:25.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:25.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:25.454+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:25.456234+0000) 2024-02-20T09:25:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:25.749+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:25.751870+0000) 2024-02-20T09:25:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:25.747+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:25.749487+0000) 2024-02-20T09:25:26.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:26.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:26.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:26.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:26.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:26.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:26.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:26.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:26.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:26.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:26.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:26.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:26.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:26.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:26.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:26.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:26.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:26.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:26.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:26.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:26.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:26.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:26.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:26.422+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:26.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:26.456+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.458653+0000) 2024-02-20T09:25:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:26.753+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.755246+0000) 2024-02-20T09:25:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:26.750+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.752159+0000) 2024-02-20T09:25:27.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:27.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:27.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:27.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:27.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:27.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:27.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:27.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:27.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:27.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:27.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:27.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:27.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:27.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:27.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:27.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:27.387+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:27.388+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:27.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:27.459+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.461228+0000) 2024-02-20T09:25:27.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:27.755+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.757441+0000) 2024-02-20T09:25:27.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:27.807+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.809176+0000) 2024-02-20T09:25:27.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:27.753+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.755299+0000) 2024-02-20T09:25:27.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:27.808+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.810880+0000) 2024-02-20T09:25:27.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:27.811+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64465 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:27.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:27.811+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.813136+0000) 2024-02-20T09:25:28.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:28.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:28.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:28.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:28.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:28.444+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:28.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:28.445+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:28.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:28.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:28.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:28.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:28.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:28.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:28.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:28.315+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:28.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:28.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:28.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:28.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:28.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:28.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:28.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:28.385+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:28.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:28.460+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:28.461447+0000) 2024-02-20T09:25:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:28.756+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:28.757626+0000) 2024-02-20T09:25:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:28.754+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:28.755503+0000) 2024-02-20T09:25:29.662 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:29.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:29.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:29.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:29.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:29.476+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:29.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:29.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:29.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:29.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:29.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:29.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:29.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:29.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:29.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:29.311+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:29.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:29.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:29.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:29.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:29.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:29.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:29.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:29.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:29.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:29.463+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:29.464564+0000) 2024-02-20T09:25:29.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:29.759+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:29.760584+0000) 2024-02-20T09:25:29.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:29.758+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:29.759020+0000) 2024-02-20T09:25:30.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:30.442+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:30.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:30.442+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:30.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:30.442+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:30.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:30.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:30.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:30.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:30.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:30.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:30.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:30.443+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:30.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:30.443+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:30.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:30.465+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:30.466867+0000) 2024-02-20T09:25:30.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:30.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:30.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:30.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:30.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:30.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:30.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:30.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:30.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:30.762+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:30.763030+0000) 2024-02-20T09:25:30.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:30.761+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:30.762127+0000) 2024-02-20T09:25:31.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:31.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:31.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:31.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:31.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:31.469+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:31.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:31.470+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:31.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:31.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:31.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:31.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:31.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:31.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:31.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:31.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:31.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:31.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:31.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:31.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:31.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:31.433+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:31.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:31.434+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:31.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:31.468+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.469754+0000) 2024-02-20T09:25:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:31.764+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.765546+0000) 2024-02-20T09:25:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:31.764+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.765434+0000) 2024-02-20T09:25:32.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:32.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:32.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:32.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:32.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:32.516+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:32.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:32.517+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:32.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:32.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:32.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:32.470+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:32.471+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.472221+0000) 2024-02-20T09:25:32.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:32.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:32.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:32.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:32.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:32.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:32.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:32.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:32.941 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:32.820+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64504 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:32.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:32.817+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.817963+0000) 2024-02-20T09:25:32.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:32.767+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.768083+0000) 2024-02-20T09:25:32.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:32.817+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.818398+0000) 2024-02-20T09:25:32.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:32.768+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.769007+0000) 2024-02-20T09:25:32.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:32.816+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.817920+0000) 2024-02-20T09:25:33.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:33.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:33.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:33.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:33.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:33.505+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:33.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:33.506+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:33.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:33.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:33.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:33.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:33.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:33.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:33.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:33.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:33.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:33.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:33.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:33.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:33.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:33.432+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:33.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:33.433+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:33.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:33.471+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:33.472485+0000) 2024-02-20T09:25:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:33.767+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:33.768287+0000) 2024-02-20T09:25:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:33.768+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:33.769323+0000) 2024-02-20T09:25:34.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:34.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:34.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:34.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:34.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:34.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:34.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:34.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:34.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:34.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:34.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:34.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:34.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:34.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:34.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:34.408+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:34.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:34.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:34.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:34.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:34.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:34.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:34.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:34.393+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:34.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:34.474+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:34.475750+0000) 2024-02-20T09:25:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:34.770+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:34.770950+0000) 2024-02-20T09:25:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:34.771+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:34.772804+0000) 2024-02-20T09:25:35.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:35.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:35.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:35.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:35.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:35.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:35.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:35.504+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:35.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:35.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:35.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:35.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:35.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:35.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:35.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:35.368+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:35.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:35.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:35.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:35.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:35.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:35.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:35.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:35.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:35.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:35.478+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:35.478954+0000) 2024-02-20T09:25:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:35.774+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:35.775732+0000) 2024-02-20T09:25:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:35.773+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:35.774207+0000) 2024-02-20T09:25:36.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:36.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:36.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:36.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:36.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:36.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:36.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:36.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:36.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:36.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:36.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:36.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:36.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:36.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:36.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:36.409+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:36.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:36.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:36.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:36.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:36.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:36.326+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:36.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:36.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:36.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:36.481+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.482291+0000) 2024-02-20T09:25:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:36.777+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.778518+0000) 2024-02-20T09:25:36.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:36.776+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.777011+0000) 2024-02-20T09:25:37.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:37.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:37.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:37.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:37.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:37.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:37.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:37.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:37.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:37.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:37.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:37.522+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:37.523+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:37.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:37.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:37.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:37.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:37.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:37.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:37.483+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.484922+0000) 2024-02-20T09:25:37.951 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:37.827+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64544 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:37.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:37.780+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.781831+0000) 2024-02-20T09:25:37.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:37.825+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.825961+0000) 2024-02-20T09:25:37.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:37.778+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.779819+0000) 2024-02-20T09:25:37.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:37.824+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.825745+0000) 2024-02-20T09:25:37.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:37.824+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.825739+0000) 2024-02-20T09:25:38.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:38.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:38.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:38.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:38.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:38.542+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:38.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:38.544+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:38.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:38.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:38.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:38.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:38.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:38.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:38.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:38.427+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:38.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:38.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:38.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:38.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:38.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:38.380+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:38.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:38.381+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:38.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:38.484+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:38.485130+0000) 2024-02-20T09:25:38.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:38.779+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:38.780038+0000) 2024-02-20T09:25:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:38.781+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:38.782025+0000) 2024-02-20T09:25:39.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:39.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:39.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:39.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:39.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:39.558+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:39.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:39.559+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:39.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:39.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:39.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:39.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:39.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:39.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:39.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:39.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:39.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:39.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:39.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:39.486+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:39.487650+0000) 2024-02-20T09:25:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:39.782+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:39.783392+0000) 2024-02-20T09:25:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:39.783+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:39.784597+0000) 2024-02-20T09:25:40.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:40.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:40.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:40.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:40.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:40.372+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:40.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:40.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:40.283+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:40.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:40.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:40.489+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:40.490051+0000) 2024-02-20T09:25:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:40.786+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:40.787534+0000) 2024-02-20T09:25:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:40.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:40.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:40.586+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:40.587+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:40.785+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:40.786908+0000) 2024-02-20T09:25:41.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:41.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:41.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:41.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:41.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:41.396+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:41.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:41.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:41.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:41.320+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:41.614 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:41.491+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.492507+0000) 2024-02-20T09:25:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:41.790+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.790958+0000) 2024-02-20T09:25:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:41.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:41.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:41.611+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:41.612+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:41.788+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.789718+0000) 2024-02-20T09:25:42.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:42.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:42.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:42.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:42.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:42.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:42.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:42.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:42.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:42.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:42.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:42.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:42.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:42.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:42.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:42.330+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:42.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:42.494+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.495161+0000) 2024-02-20T09:25:42.954 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:42.835+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64583 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:42.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:42.792+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.793333+0000) 2024-02-20T09:25:42.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:42.832+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.833513+0000) 2024-02-20T09:25:42.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:42.832+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.832990+0000) 2024-02-20T09:25:42.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:42.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:42.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:42.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:42.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:42.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:42.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:42.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:42.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:42.791+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.792805+0000) 2024-02-20T09:25:42.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:42.833+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.834557+0000) 2024-02-20T09:25:43.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:43.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:43.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:43.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:43.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:43.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:43.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:43.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:43.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:43.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:43.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:43.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:43.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:43.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:43.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:43.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:43.497 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:43.494+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:43.495363+0000) 2024-02-20T09:25:43.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:43.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:43.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:43.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:43.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:43.673+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:43.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:43.674+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:44.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:43.792+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:43.792971+0000) 2024-02-20T09:25:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:43.792+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:43.793516+0000) 2024-02-20T09:25:44.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:44.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:44.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:44.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:44.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:44.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:44.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:44.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:44.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:44.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:44.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:44.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:44.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:44.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:44.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:44.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:44.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:44.496+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:44.497909+0000) 2024-02-20T09:25:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:44.795+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:44.796027+0000) 2024-02-20T09:25:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:44.708+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:44.708+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:44.708+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:44.709+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:44.795+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:44.796401+0000) 2024-02-20T09:25:45.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:45.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:45.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:45.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:45.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:45.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:45.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:45.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:45.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:45.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:45.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:45.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:45.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:45.296+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:45.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:45.499+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:45.500505+0000) 2024-02-20T09:25:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:45.797+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:45.798715+0000) 2024-02-20T09:25:45.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:45.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:45.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:45.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:45.752+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:45.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:45.798+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:45.799629+0000) 2024-02-20T09:25:46.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:46.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:46.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:46.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:46.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:46.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:46.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:46.509+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:46.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:46.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:46.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:46.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:46.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:46.316+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:46.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:46.317+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:46.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:46.503+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.504188+0000) 2024-02-20T09:25:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:46.800+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.801672+0000) 2024-02-20T09:25:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:46.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:46.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:46.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:46.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:46.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:46.801+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.802604+0000) 2024-02-20T09:25:47.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:47.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:47.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:47.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:47.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:47.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:47.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:47.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:47.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:47.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:47.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:47.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:47.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:47.323+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:47.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:47.324+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:47.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:47.505+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.506446+0000) 2024-02-20T09:25:47.964 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:47.843+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64623 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:47.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:47.840+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.841035+0000) 2024-02-20T09:25:47.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:47.803+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.804291+0000) 2024-02-20T09:25:47.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:47.840+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.841741+0000) 2024-02-20T09:25:47.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:47.716+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:47.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:47.716+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:47.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:47.716+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:47.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:47.717+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:47.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:47.804+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.805351+0000) 2024-02-20T09:25:47.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:47.839+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.840154+0000) 2024-02-20T09:25:48.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:48.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:48.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:48.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:48.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:48.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:48.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:48.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:48.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:48.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:48.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:48.319+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:48.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:48.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:48.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:48.505+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:48.506674+0000) 2024-02-20T09:25:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:48.803+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:48.804497+0000) 2024-02-20T09:25:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:48.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:48.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:48.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:48.712+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:48.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:48.804+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:48.805523+0000) 2024-02-20T09:25:49.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:49.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:49.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:49.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:49.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:49.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:49.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:49.427+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:49.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:49.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:49.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:49.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:49.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:49.314+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:49.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:49.315+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:49.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:49.508+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:49.509167+0000) 2024-02-20T09:25:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:49.805+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:49.806685+0000) 2024-02-20T09:25:49.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:49.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:49.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:49.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:49.743+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:49.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:49.807+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:49.808182+0000) 2024-02-20T09:25:50.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:50.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:50.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:50.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:50.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:50.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:50.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:50.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:50.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:50.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:50.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:50.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:50.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:50.325+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:50.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:50.326+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:50.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:50.510+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:50.511222+0000) 2024-02-20T09:25:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:50.807+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:50.808855+0000) 2024-02-20T09:25:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:50.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:50.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:50.770+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:50.771+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:50.810+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:50.811736+0000) 2024-02-20T09:25:51.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:51.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:51.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:51.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:51.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:51.491+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:51.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:51.492+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:51.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:51.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:51.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:51.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:51.346+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:51.512+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.513319+0000) 2024-02-20T09:25:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:51.810+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.811026+0000) 2024-02-20T09:25:51.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:51.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:51.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:51.739+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:51.740+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:51.813+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.814026+0000) 2024-02-20T09:25:52.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:52.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:52.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:52.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:52.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:52.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:52.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:52.536+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:52.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:52.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:52.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:52.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:52.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:52.369+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:52.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:52.370+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:52.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:52.515+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.515956+0000) 2024-02-20T09:25:52.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:52.852+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64663 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:52.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:52.847+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.848924+0000) 2024-02-20T09:25:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:52.812+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.813113+0000) 2024-02-20T09:25:52.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:52.848+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.849518+0000) 2024-02-20T09:25:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:52.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:52.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:52.713+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:52.714+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:52.815+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.816489+0000) 2024-02-20T09:25:52.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:52.848+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.849268+0000) 2024-02-20T09:25:53.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:53.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:53.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:53.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:53.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:53.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:53.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:53.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:53.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:53.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:53.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:53.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:53.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:53.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:53.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:53.410+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:53.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:53.515+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:53.516141+0000) 2024-02-20T09:25:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:53.812+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:53.813316+0000) 2024-02-20T09:25:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:53.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:53.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:53.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:53.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:53.754+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:53.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:53.815+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:53.816681+0000) 2024-02-20T09:25:54.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:54.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:54.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:54.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:54.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:54.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:54.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:54.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:54.450+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:54.451+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:54.518+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:54.518940+0000) 2024-02-20T09:25:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:54.814+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:54.815827+0000) 2024-02-20T09:25:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:54.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:54.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:54.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:54.751+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:54.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:54.819+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:54.820017+0000) 2024-02-20T09:25:55.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:55.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:55.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:55.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:55.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:55.513+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:55.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:55.514+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:55.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:55.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:55.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:55.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:55.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:55.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:55.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:55.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:55.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:55.520+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:55.520932+0000) 2024-02-20T09:25:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:55.817+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:55.818021+0000) 2024-02-20T09:25:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:55.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:55.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:55.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:55.725+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:55.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:55.821+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:55.822506+0000) 2024-02-20T09:25:56.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:56.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:56.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:56.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:56.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:56.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:56.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:56.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:56.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:56.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:56.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:56.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:56.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:56.459+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:56.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:56.460+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:56.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:56.523+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.524573+0000) 2024-02-20T09:25:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:56.820+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.821673+0000) 2024-02-20T09:25:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:56.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:56.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:56.748+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:56.749+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:56.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:56.823+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.824716+0000) 2024-02-20T09:25:57.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:57.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:57.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:57.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:57.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:57.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:57.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:57.494+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:57.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:57.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:57.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:57.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:57.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:57.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:57.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:57.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:57.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:57.527+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.528141+0000) 2024-02-20T09:25:57.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:25:57.860+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64703 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:25:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:57.823+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.824141+0000) 2024-02-20T09:25:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:57.857+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.858040+0000) 2024-02-20T09:25:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:57.856+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.857385+0000) 2024-02-20T09:25:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:57.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:57.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:57.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:57.715+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:57.826+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.827351+0000) 2024-02-20T09:25:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:57.856+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.857055+0000) 2024-02-20T09:25:58.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:58.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:58.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:58.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:58.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:58.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:58.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:58.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:58.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:58.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:58.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:58.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:58.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:58.467+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:58.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:58.468+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:58.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:58.527+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:58.528365+0000) 2024-02-20T09:25:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:58.823+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:58.824337+0000) 2024-02-20T09:25:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:58.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:58.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:58.740+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:58.741+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:58.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:58.826+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:58.827552+0000) 2024-02-20T09:25:59.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:59.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:59.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:59.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:59.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:59.478+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:25:59.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:59.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:59.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:59.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:59.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:59.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:59.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:59.461+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:25:59.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:59.462+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:59.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:25:59.531+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:59.531951+0000) 2024-02-20T09:25:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:25:59.826+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:59.827897+0000) 2024-02-20T09:25:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:59.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:59.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:59.731+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:25:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:59.732+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:25:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:25:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:25:59.830+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:59.831070+0000) 2024-02-20T09:26:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:00.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:00.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:00.442+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:00.444+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:00.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:00.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:00.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:00.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:00.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:00.419+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:00.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:00.420+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:00.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:00.534+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:00.535284+0000) 2024-02-20T09:26:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:00.830+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:00.831255+0000) 2024-02-20T09:26:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:00.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:00.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:00.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:00.707+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:00.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:00.833+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:00.834638+0000) 2024-02-20T09:26:01.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:01.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:01.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:01.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:01.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:01.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:01.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:01.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:01.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:01.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:01.420+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:01.421+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:01.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:01.536+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.537797+0000) 2024-02-20T09:26:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:01.832+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.833480+0000) 2024-02-20T09:26:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:01.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:01.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:01.742+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:01.743+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:01.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:01.837+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.838205+0000) 2024-02-20T09:26:02.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:02.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:02.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:02.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:02.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:02.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:02.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:02.389+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:02.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:02.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:02.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:02.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:02.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:02.385+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:02.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:02.386+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:02.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:02.539+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.540059+0000) 2024-02-20T09:26:02.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:02.868+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64742 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:02.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:02.864+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.865703+0000) 2024-02-20T09:26:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:02.834+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.835490+0000) 2024-02-20T09:26:02.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:02.864+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.865289+0000) 2024-02-20T09:26:02.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:02.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:02.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:02.788+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:02.789+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:02.839+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.840461+0000) 2024-02-20T09:26:02.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:02.864+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.865139+0000) 2024-02-20T09:26:03.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:03.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:03.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:03.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:03.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:03.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:03.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:03.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:03.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:03.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:03.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:03.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:03.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:03.430+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:03.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:03.431+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:03.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:03.539+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:03.540302+0000) 2024-02-20T09:26:03.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:03.834+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:03.835740+0000) 2024-02-20T09:26:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:03.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:03.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:03.837+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:03.837+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:03.839+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:03.840677+0000) 2024-02-20T09:26:04.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:04.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:04.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:04.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:04.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:04.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:04.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:04.379+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:04.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:04.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:04.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:04.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:04.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:04.416+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:04.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:04.417+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:04.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:04.541+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:04.542196+0000) 2024-02-20T09:26:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:04.837+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:04.838126+0000) 2024-02-20T09:26:04.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:04.843+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:04.844571+0000) 2024-02-20T09:26:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:04.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:04.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:04.858+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:04.859+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:05.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:05.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:05.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:05.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:05.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:05.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:05.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:05.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:05.415+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:05.416+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:05.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:05.543+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:05.544869+0000) 2024-02-20T09:26:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:05.846+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:05.847097+0000) 2024-02-20T09:26:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:05.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:05.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:05.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:05.868+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:05.839+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:05.840733+0000) 2024-02-20T09:26:06.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:06.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:06.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:06.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:06.413+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:06.414+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:06.546+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.547308+0000) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:06.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:06.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:06.392+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:06.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:06.392+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:06.842+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.843235+0000) 2024-02-20T09:26:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:06.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:06.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:06.829+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:06.830+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:06.848+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.849642+0000) 2024-02-20T09:26:07.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:07.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:07.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:07.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:07.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:07.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:07.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:07.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:07.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:07.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:07.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:07.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:07.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:07.463+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:07.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:07.464+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:07.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:07.548+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.549446+0000) 2024-02-20T09:26:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:07.844+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.845689+0000) 2024-02-20T09:26:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:07.872+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.873861+0000) 2024-02-20T09:26:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:07.851+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.852074+0000) 2024-02-20T09:26:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:07.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:07.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:07.856+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:07.857+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:07.872+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.873890+0000) 2024-02-20T09:26:07.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:07.873+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.874334+0000) 2024-02-20T09:26:07.973 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:07.876+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64782 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:08.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:08.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:08.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:08.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:08.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:08.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:08.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:08.384+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:08.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:08.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:08.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:08.418+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:08.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:08.420+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:08.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:08.548+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:08.549683+0000) 2024-02-20T09:26:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:08.851+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:08.852322+0000) 2024-02-20T09:26:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:08.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:08.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:08.868+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:08.868+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:08.844+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:08.845926+0000) 2024-02-20T09:26:09.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:09.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:09.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:09.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:09.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:09.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:09.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:09.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:09.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:09.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:09.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:09.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:09.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:09.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:09.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:09.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:09.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:09.551+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:09.552077+0000) 2024-02-20T09:26:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:09.847+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:09.848306+0000) 2024-02-20T09:26:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:09.854+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:09.855411+0000) 2024-02-20T09:26:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:09.854+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:09.854+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:09.854+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:09.856+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:10.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:10.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:10.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:10.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:10.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:10.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:10.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:10.408+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:10.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:10.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:10.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:10.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:10.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:10.441+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:10.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:10.442+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:10.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:10.553+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:10.554489+0000) 2024-02-20T09:26:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:10.849+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:10.850909+0000) 2024-02-20T09:26:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:10.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:10.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:10.844+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:10.845+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:10.858+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:10.859217+0000) 2024-02-20T09:26:11.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:11.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:11.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:11.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:11.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:11.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:11.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:11.379+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:11.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:11.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:11.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:11.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:11.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:11.412+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:11.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:11.413+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:11.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:11.556+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:11.557036+0000) 2024-02-20T09:26:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:11.852+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:11.853365+0000) 2024-02-20T09:26:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:11.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:11.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:11.810+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:11.810+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:11.861+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:11.862042+0000) 2024-02-20T09:26:12.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:12.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:12.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:12.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:12.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:12.411+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:12.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:12.412+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:12.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:12.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:12.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:12.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:12.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:12.409+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:12.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:12.409+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:12.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:12.558+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.559617+0000) 2024-02-20T09:26:12.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:12.884+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64821 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:12.854+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.855790+0000) 2024-02-20T09:26:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:12.880+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.881237+0000) 2024-02-20T09:26:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:12.883+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.884101+0000) 2024-02-20T09:26:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:12.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:12.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:12.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:12.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:12.843+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:12.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:12.844+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:12.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:12.864+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.865179+0000) 2024-02-20T09:26:12.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:12.881+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.882759+0000) 2024-02-20T09:26:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:13.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:13.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:13.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:13.385+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:13.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:13.386+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:13.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:13.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:13.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:13.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:13.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:13.431+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:13.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:13.432+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:13.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:13.558+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:13.559848+0000) 2024-02-20T09:26:13.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:13.864+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:13.865456+0000) 2024-02-20T09:26:13.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:13.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:13.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:13.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:13.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:13.888+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:13.951 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:13.889+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:13.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:13.855+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:13.856012+0000) 2024-02-20T09:26:14.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:14.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:14.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:14.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:14.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:14.408+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:14.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:14.408+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:14.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:14.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:14.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:14.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:14.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:14.478+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:14.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:14.479+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:14.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:14.561+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:14.562025+0000) 2024-02-20T09:26:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:14.858+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:14.859614+0000) 2024-02-20T09:26:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:14.866+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:14.867561+0000) 2024-02-20T09:26:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:14.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:14.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:14.867+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:14.868+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:15.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:15.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:15.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:15.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:15.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:15.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:15.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:15.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:15.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:15.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:15.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:15.436+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:15.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:15.437+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:15.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:15.564+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:15.565394+0000) 2024-02-20T09:26:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:15.860+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:15.861727+0000) 2024-02-20T09:26:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:15.869+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:15.870377+0000) 2024-02-20T09:26:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:15.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:15.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:15.907+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:15.908+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:16.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:16.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:16.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:16.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:16.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:16.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:16.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:16.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:16.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:16.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:16.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:16.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:16.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:16.393+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:16.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:16.394+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:16.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:16.568+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:16.568947+0000) 2024-02-20T09:26:16.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:16.863+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:16.864187+0000) 2024-02-20T09:26:16.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:16.872+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:16.872966+0000) 2024-02-20T09:26:16.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:16.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:16.939 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:16.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:16.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:17.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:16.937+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:17.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:17.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:17.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:17.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:17.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:17.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:17.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:17.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:17.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:17.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:17.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:17.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:17.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:17.404+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:17.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:17.404+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:17.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:17.570+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.571368+0000) 2024-02-20T09:26:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:17.873+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.875440+0000) 2024-02-20T09:26:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:17.887+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.889665+0000) 2024-02-20T09:26:17.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:17.864+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.866500+0000) 2024-02-20T09:26:17.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:17.889+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.891236+0000) 2024-02-20T09:26:17.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:17.891+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64861 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:17.887+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.889423+0000) 2024-02-20T09:26:18.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:17.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:18.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:17.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:18.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:17.968+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:18.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:17.969+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:18.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:18.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:18.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:18.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:18.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:18.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:18.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:18.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:18.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:18.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:18.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:18.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:18.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:18.451+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:18.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:18.451+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:18.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:18.569+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:18.571605+0000) 2024-02-20T09:26:18.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:18.864+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:18.866778+0000) 2024-02-20T09:26:18.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:18.873+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:18.875666+0000) 2024-02-20T09:26:18.940 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:18.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:18.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:18.937+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:19.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:18.938+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:19.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:19.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:19.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:19.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:19.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:19.512+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:19.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:19.513+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:19.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:19.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:19.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:19.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:19.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:19.424+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:19.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:19.425+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:19.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:19.572+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:19.574021+0000) 2024-02-20T09:26:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:19.867+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:19.869251+0000) 2024-02-20T09:26:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:19.875+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:19.877867+0000) 2024-02-20T09:26:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:19.892+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:19.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:19.893+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:19.893+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:20.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:20.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:20.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:20.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:20.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:20.509+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:20.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:20.510+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:20.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:20.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:20.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:20.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:20.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:20.403+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:20.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:20.405+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:20.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:20.575+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:20.577680+0000) 2024-02-20T09:26:20.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:20.869+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:20.871538+0000) 2024-02-20T09:26:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:20.878+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:20.880491+0000) 2024-02-20T09:26:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:20.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:20.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:20.887+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:20.887+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:21.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:21.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:21.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:21.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:21.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:21.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:21.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:21.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:21.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:21.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:21.454+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:21.455+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:21.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:21.579+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:21.580673+0000) 2024-02-20T09:26:21.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:21.873+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:21.874345+0000) 2024-02-20T09:26:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:21.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:21.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:21.841+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:21.841+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:21.882+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:21.883682+0000) 2024-02-20T09:26:22.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:22.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:22.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:22.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:22.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:22.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:22.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:22.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:22.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:22.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:22.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:22.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:22.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:22.458+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:22.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:22.459+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:22.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:22.582+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.583146+0000) 2024-02-20T09:26:22.918 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:22.901+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64900 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:22.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:22.896+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.897685+0000) 2024-02-20T09:26:22.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:22.876+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.877020+0000) 2024-02-20T09:26:22.919 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:22.897+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.898330+0000) 2024-02-20T09:26:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:22.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:22.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:22.812+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:22.813+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:22.885+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.886151+0000) 2024-02-20T09:26:22.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:22.896+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.897877+0000) 2024-02-20T09:26:23.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:23.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:23.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:23.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:23.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:23.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:23.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:23.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:23.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:23.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:23.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:23.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:23.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:23.496+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:23.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:23.497+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:23.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:23.582+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:23.583422+0000) 2024-02-20T09:26:23.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:23.876+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:23.877424+0000) 2024-02-20T09:26:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:23.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:23.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:23.823+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:23.824+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:23.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:23.885+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:23.886384+0000) 2024-02-20T09:26:24.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:24.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:24.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:24.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:24.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:24.396+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:24.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:24.397+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:24.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:24.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:24.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:24.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:24.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:24.470+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:24.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:24.471+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:24.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:24.585+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:24.586058+0000) 2024-02-20T09:26:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:24.878+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:24.879920+0000) 2024-02-20T09:26:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:24.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:24.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:24.795+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:24.796+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:24.887+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:24.888769+0000) 2024-02-20T09:26:25.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:25.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:25.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:25.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:25.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:25.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:25.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:25.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:25.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:25.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:25.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:25.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:25.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:25.422+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:25.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:25.423+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:25.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:25.587+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:25.588526+0000) 2024-02-20T09:26:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:25.881+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:25.882362+0000) 2024-02-20T09:26:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:25.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:25.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:25.794+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:25.795+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:25.890+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:25.891412+0000) 2024-02-20T09:26:26.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:26.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:26.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:26.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:26.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:26.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:26.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:26.354+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:26.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:26.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:26.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:26.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:26.464+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:26.465+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:26.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:26.590+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:26.590970+0000) 2024-02-20T09:26:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:26.883+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:26.884825+0000) 2024-02-20T09:26:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:26.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:26.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:26.797+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:26.797+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:26.893+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:26.894103+0000) 2024-02-20T09:26:27.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:27.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:27.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:27.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:27.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:27.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:27.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:27.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:27.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:27.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:27.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:27.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:27.595 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:27.428+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:27.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:27.429+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:27.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:27.759+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:27.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:27.759+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:27.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:27.759+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:27.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:27.759+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:27.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:27.593+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.593959+0000) 2024-02-20T09:26:28.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:27.908+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64940 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:27.906+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.907083+0000) 2024-02-20T09:26:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:27.895+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.896185+0000) 2024-02-20T09:26:28.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:27.905+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.906191+0000) 2024-02-20T09:26:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:27.886+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.887028+0000) 2024-02-20T09:26:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:27.904+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.905707+0000) 2024-02-20T09:26:28.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:28.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:28.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:28.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:28.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:28.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:28.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:28.315+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:28.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:28.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:28.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:28.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:28.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:28.452+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:28.596 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:28.453+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:28.888 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:28.593+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:28.594235+0000) 2024-02-20T09:26:28.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:28.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:28.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:28.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:28.889 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:28.750+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:28.889 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:28.751+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:29.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:28.886+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:28.887318+0000) 2024-02-20T09:26:29.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:28.895+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:28.896517+0000) 2024-02-20T09:26:29.598 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:29.281+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:29.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:29.281+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:29.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:29.281+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:29.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:29.283+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:29.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:29.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:29.599 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:29.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:29.600 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:29.448+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:29.600 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:29.449+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:29.600 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:29.596+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:29.597663+0000) 2024-02-20T09:26:29.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:29.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:29.891 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:29.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:29.891 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:29.790+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:29.891 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:29.791+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:30.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:29.899+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:29.900053+0000) 2024-02-20T09:26:30.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:29.888+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:29.889829+0000) 2024-02-20T09:26:30.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:30.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:30.726 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:30.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:30.726 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:30.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:30.726 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:30.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:30.726 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:30.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:30.726 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:30.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:30.727 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:30.476+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:30.727 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:30.476+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:30.893 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:30.599+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:30.600298+0000) 2024-02-20T09:26:30.893 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:30.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:30.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:30.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:30.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:30.779+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:30.894 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:30.780+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:31.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:30.891+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:30.892255+0000) 2024-02-20T09:26:31.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:30.901+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:30.902148+0000) 2024-02-20T09:26:31.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:31.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:31.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:31.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:31.604 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:31.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:31.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:31.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:31.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:31.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:31.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:31.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:31.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:31.491+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:31.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:31.492+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:31.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:31.601+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:31.602801+0000) 2024-02-20T09:26:31.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:31.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:31.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:31.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:31.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:31.736+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:31.896 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:31.737+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:31.904+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:31.905115+0000) 2024-02-20T09:26:32.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:31.893+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:31.894853+0000) 2024-02-20T09:26:32.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:32.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:32.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:32.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:32.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:32.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:32.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:32.295+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:32.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:32.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:32.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:32.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:32.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:32.536+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:32.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:32.537+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:32.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:32.705+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:32.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:32.705+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:32.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:32.705+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:32.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:32.705+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:32.899 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:32.605+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.605957+0000) 2024-02-20T09:26:33.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:32.918+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 64981 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:32.913+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.913939+0000) 2024-02-20T09:26:33.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:32.907+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.908724+0000) 2024-02-20T09:26:33.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:32.912+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.913773+0000) 2024-02-20T09:26:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:32.896+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.897209+0000) 2024-02-20T09:26:33.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:32.913+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.914358+0000) 2024-02-20T09:26:33.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:33.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:33.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:33.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:33.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:33.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:33.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:33.293+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:33.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:33.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:33.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:33.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:33.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:33.504+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:33.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:33.505+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:33.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:33.605+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:33.606272+0000) 2024-02-20T09:26:33.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:33.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:33.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:33.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:33.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:33.681+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:33.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:33.682+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:33.908+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:33.909059+0000) 2024-02-20T09:26:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:33.896+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:33.897554+0000) 2024-02-20T09:26:34.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:34.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:34.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:34.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:34.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:34.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:34.610 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:34.534+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:34.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:34.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:34.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:34.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:34.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:34.327+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:34.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:34.328+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:34.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:34.607+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:34.608749+0000) 2024-02-20T09:26:34.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:34.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:34.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:34.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:34.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:34.682+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:34.902 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:34.683+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:35.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:34.910+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:34.911925+0000) 2024-02-20T09:26:35.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:34.899+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:34.900506+0000) 2024-02-20T09:26:35.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:35.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:35.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:35.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:35.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:35.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:35.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:35.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:35.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:35.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:35.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:35.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:35.615 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:35.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:35.616 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:35.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:35.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:35.610+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:35.611066+0000) 2024-02-20T09:26:35.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:35.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:35.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:35.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:35.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:35.687+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:35.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:35.687+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:36.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:35.913+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:35.914099+0000) 2024-02-20T09:26:36.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:35.901+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:35.902462+0000) 2024-02-20T09:26:36.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:36.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:36.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:36.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:36.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:36.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:36.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:36.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:36.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:36.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:36.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:36.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:36.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:36.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:36.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:36.693+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:36.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:36.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:36.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:36.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:36.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:36.544+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:36.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:36.546+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:36.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:36.613+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:36.613929+0000) 2024-02-20T09:26:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:36.915+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:36.916529+0000) 2024-02-20T09:26:37.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:36.904+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:36.905023+0000) 2024-02-20T09:26:37.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:37.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:37.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:37.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:37.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:37.250+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:37.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:37.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:37.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:37.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:37.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:37.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:37.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:37.677+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:37.909 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:37.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:37.909 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:37.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:37.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:37.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:37.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:37.539+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:37.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:37.616+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.617549+0000) 2024-02-20T09:26:38.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:37.927+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65021 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:37.922+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.923858+0000) 2024-02-20T09:26:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:37.917+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.918628+0000) 2024-02-20T09:26:38.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:37.924+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.925485+0000) 2024-02-20T09:26:38.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:37.906+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.907820+0000) 2024-02-20T09:26:38.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:37.923+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.924860+0000) 2024-02-20T09:26:38.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:38.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:38.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:38.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:38.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:38.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:38.582 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:38.253+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:38.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:38.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:38.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:38.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:38.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:38.642+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:38.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:38.643+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:38.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:38.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:38.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:38.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:38.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:38.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:38.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:38.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:38.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:38.616+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:38.617893+0000) 2024-02-20T09:26:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:38.917+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:38.918914+0000) 2024-02-20T09:26:39.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:38.907+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:38.908181+0000) 2024-02-20T09:26:39.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:39.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:39.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:39.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:39.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:39.264+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:39.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:39.265+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:39.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:39.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:39.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:39.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:39.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:39.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:39.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:39.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:39.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:39.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:39.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:39.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:39.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:39.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:39.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:39.533+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:39.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:39.619+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:39.620450+0000) 2024-02-20T09:26:40.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:39.920+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:39.921105+0000) 2024-02-20T09:26:40.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:39.909+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:39.910733+0000) 2024-02-20T09:26:40.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:40.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:40.558 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:40.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:40.559 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:40.257+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:40.559 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:40.258+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:40.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:40.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:40.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:40.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:40.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:40.578+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:40.914 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:40.579+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:40.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:40.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:40.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:40.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:40.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:40.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:40.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:40.557+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:40.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:40.623+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:40.623951+0000) 2024-02-20T09:26:41.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:40.923+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:40.924382+0000) 2024-02-20T09:26:41.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:40.912+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:40.913081+0000) 2024-02-20T09:26:41.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:41.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:41.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:41.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:41.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:41.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:41.557 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:41.287+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:41.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:41.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:41.917 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:41.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:41.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:41.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:41.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:41.555+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:41.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:41.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:41.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:41.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:41.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:41.597+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:41.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:41.598+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:41.919 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:41.626+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:41.627270+0000) 2024-02-20T09:26:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:41.926+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:41.927737+0000) 2024-02-20T09:26:42.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:41.915+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:41.916496+0000) 2024-02-20T09:26:42.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:42.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:42.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:42.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:42.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:42.292+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:42.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:42.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:42.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:42.573+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:42.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:42.573+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:42.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:42.573+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:42.921 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:42.574+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:42.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:42.629+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.630879+0000) 2024-02-20T09:26:42.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:42.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:42.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:42.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:42.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:42.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:42.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:42.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:43.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:42.936+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65060 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:43.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:42.932+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.933044+0000) 2024-02-20T09:26:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:42.932+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.932954+0000) 2024-02-20T09:26:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:42.932+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.933075+0000) 2024-02-20T09:26:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:42.918+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.919840+0000) 2024-02-20T09:26:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:42.931+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.932657+0000) 2024-02-20T09:26:43.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:43.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:43.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:43.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:43.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:43.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:43.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:43.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:43.855 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:43.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:43.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:43.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:43.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:43.597+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:43.856 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:43.598+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:43.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:43.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:43.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:43.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:43.857 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:43.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:43.857 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:43.605+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:43.857 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:43.630+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:43.631174+0000) 2024-02-20T09:26:44.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:43.934+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:43.935599+0000) 2024-02-20T09:26:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:43.919+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:43.920168+0000) 2024-02-20T09:26:44.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:44.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:44.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:44.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:44.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:44.307+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:44.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:44.307+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:44.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:44.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:44.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:44.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:44.923 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:44.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:44.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:44.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:44.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:44.632+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:44.633138+0000) 2024-02-20T09:26:44.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:44.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:44.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:44.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:44.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:44.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:44.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:44.637+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:44.938+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:44.939080+0000) 2024-02-20T09:26:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:44.921+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:44.922356+0000) 2024-02-20T09:26:45.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:45.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:45.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:45.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:45.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:45.320+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:45.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:45.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:45.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:45.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:45.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:45.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:45.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:45.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:45.926 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:45.616+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:45.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:45.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:45.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:45.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:45.927 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:45.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:45.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:45.622+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:45.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:45.635+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:45.636609+0000) 2024-02-20T09:26:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:45.941+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:45.942433+0000) 2024-02-20T09:26:46.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:45.923+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:45.924819+0000) 2024-02-20T09:26:46.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:46.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:46.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:46.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:46.597 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:46.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:46.597 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:46.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:46.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:46.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:46.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:46.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:46.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:46.595+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:46.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:46.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:46.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:46.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:46.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:46.638+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:46.639295+0000) 2024-02-20T09:26:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:46.944+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:46.944948+0000) 2024-02-20T09:26:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:46.926+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:46.927606+0000) 2024-02-20T09:26:47.611 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:47.572+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:47.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:47.572+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:47.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:47.572+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:47.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:47.573+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:47.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:47.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:47.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:47.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:47.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:47.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:47.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:47.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:47.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:47.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:47.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:47.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:47.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:47.609+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:47.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:47.610+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:47.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:47.641+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.642662+0000) 2024-02-20T09:26:48.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:47.944+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65100 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:48.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:47.940+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.941068+0000) 2024-02-20T09:26:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:47.941+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.942433+0000) 2024-02-20T09:26:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:47.944+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.945327+0000) 2024-02-20T09:26:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:47.928+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.929849+0000) 2024-02-20T09:26:48.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:47.943+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.944153+0000) 2024-02-20T09:26:48.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:48.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:48.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:48.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:48.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:48.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:48.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:48.551+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:48.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:48.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:48.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:48.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:48.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:48.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:48.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:48.359+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:48.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:48.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:48.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:48.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:48.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:48.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:48.931 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:48.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:48.932 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:48.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:48.643032+0000) 2024-02-20T09:26:49.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:48.947+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:48.947973+0000) 2024-02-20T09:26:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:48.929+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:48.930257+0000) 2024-02-20T09:26:49.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:49.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:49.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:49.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:49.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:49.523+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:49.631 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:49.524+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:49.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:49.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:49.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:49.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:49.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:49.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:49.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:49.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:49.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:49.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:49.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:49.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:49.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:49.629+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:49.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:49.630+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:49.935 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:49.644+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:49.645458+0000) 2024-02-20T09:26:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:49.949+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:49.950123+0000) 2024-02-20T09:26:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:49.932+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:49.933795+0000) 2024-02-20T09:26:50.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:50.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:50.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:50.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:50.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:50.282+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:50.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:50.282+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:50.815 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:50.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:50.815 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:50.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:50.815 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:50.562+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:50.816 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:50.563+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:50.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:50.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:50.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:50.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:50.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:50.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:50.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:50.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:50.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:50.647+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:50.648810+0000) 2024-02-20T09:26:51.087 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:50.951+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:50.952142+0000) 2024-02-20T09:26:51.087 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:50.935+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:50.935945+0000) 2024-02-20T09:26:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:51.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:51.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:51.236+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:51.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:51.237+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:51.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:51.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:51.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:51.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:51.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:51.630 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:51.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:51.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:51.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:51.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:51.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:51.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:51.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:51.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:51.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:51.939 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:51.651+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:51.652456+0000) 2024-02-20T09:26:52.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:51.954+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:51.955585+0000) 2024-02-20T09:26:52.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:51.937+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:51.938757+0000) 2024-02-20T09:26:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:52.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:52.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:52.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:52.218+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:52.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:52.219+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:52.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:52.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:52.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:52.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:52.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:52.626+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:52.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:52.627+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:52.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:52.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:52.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:52.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:52.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:52.604+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:52.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:52.605+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:52.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:52.654+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.655753+0000) 2024-02-20T09:26:53.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:52.953+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65139 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:53.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:52.948+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.949617+0000) 2024-02-20T09:26:53.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:52.951+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.952146+0000) 2024-02-20T09:26:53.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:52.955+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.955959+0000) 2024-02-20T09:26:53.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:52.941+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.941928+0000) 2024-02-20T09:26:53.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:52.949+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.950133+0000) 2024-02-20T09:26:53.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:53.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:53.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:53.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:53.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:53.196+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:53.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:53.197+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:53.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:53.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:53.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:53.582+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:53.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:53.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:53.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:53.579+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:53.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:53.655+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:53.656164+0000) 2024-02-20T09:26:54.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:53.957+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:53.958533+0000) 2024-02-20T09:26:54.158 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:53.941+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:53.942350+0000) 2024-02-20T09:26:54.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:54.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:54.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:54.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:54.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:54.155+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:54.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:54.157+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:54.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:54.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:54.540+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:54.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:54.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:54.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:54.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:54.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:54.658+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:54.659488+0000) 2024-02-20T09:26:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:54.960+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:54.961137+0000) 2024-02-20T09:26:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:54.944+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:54.945049+0000) 2024-02-20T09:26:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:55.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:55.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:55.181+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:55.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:55.182+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:55.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:55.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:55.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:55.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:55.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:55.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:55.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:55.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:55.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:55.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:55.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:55.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:55.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:55.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:55.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:55.546+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:55.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:55.662+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:55.662932+0000) 2024-02-20T09:26:56.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:55.962+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:55.963148+0000) 2024-02-20T09:26:56.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:55.947+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:55.948608+0000) 2024-02-20T09:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:56.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:56.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:56.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:56.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:56.204+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:56.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:56.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:56.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:56.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:56.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:56.592+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:56.952 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:56.593+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:56.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:56.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:56.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:56.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:56.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:56.542+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:56.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:56.543+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:56.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:56.664+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:56.665582+0000) 2024-02-20T09:26:57.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:56.964+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:56.965421+0000) 2024-02-20T09:26:57.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:56.949+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:56.950925+0000) 2024-02-20T09:26:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:57.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:57.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:57.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:57.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:57.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:57.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:57.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:57.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:57.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:57.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:57.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:57.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:57.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:57.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:57.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:57.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:57.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:57.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:57.510+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:57.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:57.511+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:57.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:57.667+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.668021+0000) 2024-02-20T09:26:58.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:26:57.961+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65179 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:26:58.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:57.959+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.960321+0000) 2024-02-20T09:26:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:57.958+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.959625+0000) 2024-02-20T09:26:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:57.965+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.966136+0000) 2024-02-20T09:26:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:57.952+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.953651+0000) 2024-02-20T09:26:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:57.958+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.959576+0000) 2024-02-20T09:26:58.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:58.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:58.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:58.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:58.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:58.249+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:58.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:58.250+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:58.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:58.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:58.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:58.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:58.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:58.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:58.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:58.586+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:58.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:58.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:58.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:58.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:58.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:58.519+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:58.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:58.520+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:58.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:58.667+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:58.668452+0000) 2024-02-20T09:26:59.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:58.968+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:58.968960+0000) 2024-02-20T09:26:59.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:58.953+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:58.954044+0000) 2024-02-20T09:26:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:59.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:59.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:59.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:26:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:59.204+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:59.958 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:59.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:59.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:59.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:59.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:59.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:26:59.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:59.590+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:59.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:59.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:59.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:59.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:59.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:59.555+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:26:59.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:59.556+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:26:59.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:26:59.669+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:59.670607+0000) 2024-02-20T09:27:00.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:26:59.970+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:59.971116+0000) 2024-02-20T09:27:00.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:26:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:26:59.956+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:59.957699+0000) 2024-02-20T09:27:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:00.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:00.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:00.205+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:00.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:00.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:00.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:00.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:00.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:00.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:00.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:00.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:00.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:00.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:00.672+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:00.673810+0000) 2024-02-20T09:27:00.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:00.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:00.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:00.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:00.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:00.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:00.973+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:00.974585+0000) 2024-02-20T09:27:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:00.959+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:00.960739+0000) 2024-02-20T09:27:01.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:01.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:01.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:01.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:01.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:01.239+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:01.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:01.240+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:01.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:01.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:01.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:01.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:01.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:01.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:01.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:01.660+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:01.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:01.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:01.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:01.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:01.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:01.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:01.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:01.629+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:01.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:01.676+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:01.677491+0000) 2024-02-20T09:27:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:01.977+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:01.977931+0000) 2024-02-20T09:27:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:01.963+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:01.964266+0000) 2024-02-20T09:27:02.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:02.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:02.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:02.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:02.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:02.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:02.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:02.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:02.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:02.965 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:02.638+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:02.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:02.639+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:02.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:02.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:02.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:02.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:02.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:02.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:02.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:02.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:02.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:02.679+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.680479+0000) 2024-02-20T09:27:03.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:02.970+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65218 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:02.967+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.968138+0000) 2024-02-20T09:27:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:02.966+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.966993+0000) 2024-02-20T09:27:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:02.977+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.978350+0000) 2024-02-20T09:27:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:02.966+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.967487+0000) 2024-02-20T09:27:03.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:02.966+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.967702+0000) 2024-02-20T09:27:03.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:03.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:03.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:03.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:03.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:03.201+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:03.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:03.202+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:03.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:03.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:03.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:03.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:03.660+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:03.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:03.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:03.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:03.660+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:03.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:03.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:03.679+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:03.680782+0000) 2024-02-20T09:27:04.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:03.980+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:03.981477+0000) 2024-02-20T09:27:04.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:03.970+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:03.971258+0000) 2024-02-20T09:27:04.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:04.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:04.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:04.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:04.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:04.215+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:04.351 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:04.216+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:04.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:04.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:04.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:04.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:04.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:04.640+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:04.640+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:04.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:04.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:04.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:04.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:04.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:04.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:04.683+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:04.684360+0000) 2024-02-20T09:27:05.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:04.982+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:04.983910+0000) 2024-02-20T09:27:05.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:04.972+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:04.973683+0000) 2024-02-20T09:27:05.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:05.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:05.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:05.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:05.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:05.253+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:05.614 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:05.254+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:05.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:05.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:05.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:05.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:05.614+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:05.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:05.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:05.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:05.654+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:05.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:05.686+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:05.687145+0000) 2024-02-20T09:27:06.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:05.985+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:05.986186+0000) 2024-02-20T09:27:06.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:05.976+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:05.977284+0000) 2024-02-20T09:27:06.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:06.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:06.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:06.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:06.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:06.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:06.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:06.264+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:06.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:06.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:06.654+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:06.656+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:06.688+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:06.689704+0000) 2024-02-20T09:27:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:06.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:06.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:06.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:06.704+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:07.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:06.988+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:06.989556+0000) 2024-02-20T09:27:07.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:06.978+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:06.979687+0000) 2024-02-20T09:27:07.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:07.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:07.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:07.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:07.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:07.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:07.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:07.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:07.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:07.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:07.683+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:07.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:07.684+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:07.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:07.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:07.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:07.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:07.691+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.692382+0000) 2024-02-20T09:27:08.301 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:07.976+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.977038+0000) 2024-02-20T09:27:08.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:07.989+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.989962+0000) 2024-02-20T09:27:08.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:07.974+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.975724+0000) 2024-02-20T09:27:08.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:07.979+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.980026+0000) 2024-02-20T09:27:08.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:07.980+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65257 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:08.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:07.975+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.976375+0000) 2024-02-20T09:27:08.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:08.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:08.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:08.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:08.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:08.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:08.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:08.300+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:08.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:08.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:08.693+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:08.694+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:08.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:08.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:08.668+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:08.669+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:08.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:08.691+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:08.692836+0000) 2024-02-20T09:27:09.337 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:08.991+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:08.992764+0000) 2024-02-20T09:27:09.338 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:08.982+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:08.982959+0000) 2024-02-20T09:27:09.338 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:09.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:09.338 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:09.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:09.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:09.336+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:09.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:09.337+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:09.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:09.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:09.686+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:09.687+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:09.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:09.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:09.670+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:09.671+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:09.694+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:09.695356+0000) 2024-02-20T09:27:10.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:09.994+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:09.995309+0000) 2024-02-20T09:27:10.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:09.984+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:09.985412+0000) 2024-02-20T09:27:10.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:10.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:10.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:10.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:10.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:10.288+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:10.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:10.289+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:10.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:10.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:10.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:10.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:10.725+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:10.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:10.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:10.669+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:10.670+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:10.696+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:10.697899+0000) 2024-02-20T09:27:11.263 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:10.996+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:10.998902+0000) 2024-02-20T09:27:11.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:10.986+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:10.987954+0000) 2024-02-20T09:27:11.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:11.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:11.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:11.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:11.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:11.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:11.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:11.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:11.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:11.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:11.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:11.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:11.748+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:11.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:11.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:11.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:11.631+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:11.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:11.699+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:11.701401+0000) 2024-02-20T09:27:12.301 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:11.999+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.000970+0000) 2024-02-20T09:27:12.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:11.989+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:11.991569+0000) 2024-02-20T09:27:12.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:12.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:12.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:12.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:12.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:12.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:12.610 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:12.299+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:12.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:12.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:12.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:12.778+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:12.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:12.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:12.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:12.607+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:12.701+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.703822+0000) 2024-02-20T09:27:13.343 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:12.987+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65297 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:13.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:12.983+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.984953+0000) 2024-02-20T09:27:13.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:12.999+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:13.001300+0000) 2024-02-20T09:27:13.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:12.983+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.985122+0000) 2024-02-20T09:27:13.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:12.990+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.991989+0000) 2024-02-20T09:27:13.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:13.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:13.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:12.984+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.986316+0000) 2024-02-20T09:27:13.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:13.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:13.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:13.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:13.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:13.341+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:13.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:13.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:13.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:13.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:13.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:13.572+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:13.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:13.573+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:13.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:13.702+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:13.704272+0000) 2024-02-20T09:27:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:13.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:13.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:13.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:13.753+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:13.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:13.753+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:14.351 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:14.001+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:14.003734+0000) 2024-02-20T09:27:14.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:13.993+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:13.995714+0000) 2024-02-20T09:27:14.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:14.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:14.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:14.353+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:14.353+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:14.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:14.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:14.563+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:14.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:14.563+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:14.706+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:14.707608+0000) 2024-02-20T09:27:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:14.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:14.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:14.747+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:14.748+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:15.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:15.006+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:15.007056+0000) 2024-02-20T09:27:15.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:14.998+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:14.999092+0000) 2024-02-20T09:27:15.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:15.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:15.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:15.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:15.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:15.343+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:15.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:15.344+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:15.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:15.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:15.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:15.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:15.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:15.541+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:15.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:15.542+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:15.709+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:15.710104+0000) 2024-02-20T09:27:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:15.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:15.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:15.792+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:15.793+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:16.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:16.009+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:16.010013+0000) 2024-02-20T09:27:16.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:16.000+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:16.001638+0000) 2024-02-20T09:27:16.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:16.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:16.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:16.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:16.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:16.383+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:16.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:16.385+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:16.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:16.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:16.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:16.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:16.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:16.499+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:16.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:16.500+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:16.711+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:16.712542+0000) 2024-02-20T09:27:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:16.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:16.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:16.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:16.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:16.744+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:17.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:17.011+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.012368+0000) 2024-02-20T09:27:17.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:17.003+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.004139+0000) 2024-02-20T09:27:17.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:17.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:17.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:17.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:17.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:17.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:17.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:17.399+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:17.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:17.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:17.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:17.532+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:17.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:17.533+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:17.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:17.533+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:17.715+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.716321+0000) 2024-02-20T09:27:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:17.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:17.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:17.777+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:17.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:17.778+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:18.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:17.993+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.994836+0000) 2024-02-20T09:27:18.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:18.011+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:18.012833+0000) 2024-02-20T09:27:18.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:17.993+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.993948+0000) 2024-02-20T09:27:18.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:18.003+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:18.004603+0000) 2024-02-20T09:27:18.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:17.996+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65336 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:18.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:17.993+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.994865+0000) 2024-02-20T09:27:18.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:18.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:18.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:18.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:18.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:18.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:18.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:18.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:18.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:18.528+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:18.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:18.528+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:18.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:18.528+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:18.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:18.529+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:18.715+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:18.716708+0000) 2024-02-20T09:27:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:18.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:18.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:18.784+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:18.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:18.785+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:19.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:19.014+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:19.015194+0000) 2024-02-20T09:27:19.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:19.007+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:19.008119+0000) 2024-02-20T09:27:19.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:19.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:19.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:19.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:19.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:19.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:19.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:19.425+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:19.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:19.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:19.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:19.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:19.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:19.497+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:19.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:19.498+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:19.718+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:19.719372+0000) 2024-02-20T09:27:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:19.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:19.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:19.826+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:19.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:19.827+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:20.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:20.016+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:20.017777+0000) 2024-02-20T09:27:20.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:20.009+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:20.010561+0000) 2024-02-20T09:27:20.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:20.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:20.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:20.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:20.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:20.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:20.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:20.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:20.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:20.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:20.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:20.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:20.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:20.518+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:20.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:20.519+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:20.721+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:20.721955+0000) 2024-02-20T09:27:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:20.787+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:20.787+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:20.787+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:20.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:20.788+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:21.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:21.019+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:21.020265+0000) 2024-02-20T09:27:21.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:21.012+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:21.013311+0000) 2024-02-20T09:27:21.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:21.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:21.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:21.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:21.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:21.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:21.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:21.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:21.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:21.526+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:21.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:21.526+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:21.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:21.526+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:21.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:21.527+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:21.723+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:21.724364+0000) 2024-02-20T09:27:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:21.768+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:21.768+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:21.768+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:21.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:21.769+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:22.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:22.022+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.023710+0000) 2024-02-20T09:27:22.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:22.015+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.016765+0000) 2024-02-20T09:27:22.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:22.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:22.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:22.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:22.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:22.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:22.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:22.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:22.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:22.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:22.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:22.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:22.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:22.529+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:22.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:22.530+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:22.726+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.727011+0000) 2024-02-20T09:27:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:22.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:22.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:22.783+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:22.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:22.784+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:23.352 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:23.005+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65377 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:23.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:23.001+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.002387+0000) 2024-02-20T09:27:23.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:23.001+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.002044+0000) 2024-02-20T09:27:23.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:23.023+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.024288+0000) 2024-02-20T09:27:23.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:23.001+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.002573+0000) 2024-02-20T09:27:23.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:23.016+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.017371+0000) 2024-02-20T09:27:23.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:23.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:23.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:23.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:23.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:23.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:23.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:23.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:23.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:23.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:23.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:23.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:23.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:23.484+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:23.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:23.485+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:23.726+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.727411+0000) 2024-02-20T09:27:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:23.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:23.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:23.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:23.757+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:24.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:24.025+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:24.026852+0000) 2024-02-20T09:27:24.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:24.018+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:24.019432+0000) 2024-02-20T09:27:24.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:24.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:24.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:24.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:24.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:24.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:24.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:24.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:24.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:24.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:24.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:24.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:24.608 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:24.469+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:24.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:24.470+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:24.728+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:24.729386+0000) 2024-02-20T09:27:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:24.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:24.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:24.754+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:24.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:24.754+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:25.352 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:25.021+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:25.021951+0000) 2024-02-20T09:27:25.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:25.028+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:25.029438+0000) 2024-02-20T09:27:25.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:25.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:25.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:25.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:25.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:25.471+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:25.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:25.472+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:25.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:25.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:25.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:25.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:25.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:25.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:25.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:25.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:25.731+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:25.732745+0000) 2024-02-20T09:27:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:25.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:25.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:25.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:25.725+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:26.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:26.032+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:26.032972+0000) 2024-02-20T09:27:26.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:26.024+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:26.025562+0000) 2024-02-20T09:27:26.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:26.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:26.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:26.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:26.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:26.459+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:26.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:26.460+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:26.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:26.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:26.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:26.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:26.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:26.502+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:26.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:26.503+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:26.735+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:26.736047+0000) 2024-02-20T09:27:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:26.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:26.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:26.696+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:26.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:26.697+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:27.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:27.034+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.035623+0000) 2024-02-20T09:27:27.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:27.026+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.027687+0000) 2024-02-20T09:27:27.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:27.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:27.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:27.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:27.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:27.538+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:27.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:27.539+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:27.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:27.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:27.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:27.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:27.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:27.433+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:27.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:27.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:27.738+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.739403+0000) 2024-02-20T09:27:27.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:27.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:27.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:27.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:27.718+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:27.719+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:28.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:28.013+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65416 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:28.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:28.010+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.011204+0000) 2024-02-20T09:27:28.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:28.010+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.011628+0000) 2024-02-20T09:27:28.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:28.035+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.035952+0000) 2024-02-20T09:27:28.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:28.010+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.011401+0000) 2024-02-20T09:27:28.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:28.027+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.028079+0000) 2024-02-20T09:27:28.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:28.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:28.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:28.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:28.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:28.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:28.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:28.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:28.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:28.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:28.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:28.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:28.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:28.547+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:28.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:28.548+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:28.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:28.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:28.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:28.688+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:28.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:28.689+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:28.738+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.739881+0000) 2024-02-20T09:27:29.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:29.037+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:29.038566+0000) 2024-02-20T09:27:29.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:29.029+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:29.030551+0000) 2024-02-20T09:27:29.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:29.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:29.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:29.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:29.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:29.465+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:29.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:29.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:29.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:29.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:29.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:29.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:29.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:29.552+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:29.668 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:29.553+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:29.742+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:29.743059+0000) 2024-02-20T09:27:29.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:29.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:29.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:29.665+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:29.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:29.666+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:30.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:30.039+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:30.040860+0000) 2024-02-20T09:27:30.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:30.032+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:30.033315+0000) 2024-02-20T09:27:30.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:30.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:30.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:30.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:30.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:30.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:30.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:30.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:30.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:30.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:30.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:30.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:30.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:30.546+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:30.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:30.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:30.744+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:30.745503+0000) 2024-02-20T09:27:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:30.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:30.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:30.714+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:30.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:30.714+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:31.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:31.042+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:31.043832+0000) 2024-02-20T09:27:31.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:31.034+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:31.035551+0000) 2024-02-20T09:27:31.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:31.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:31.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:31.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:31.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:31.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:31.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:31.403+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:31.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:31.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:31.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:31.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:31.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:31.568+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:31.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:31.569+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:31.746+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:31.747697+0000) 2024-02-20T09:27:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:31.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:31.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:31.702+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:31.703+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:32.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:32.046+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.047202+0000) 2024-02-20T09:27:32.353 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:32.037+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.038847+0000) 2024-02-20T09:27:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:32.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:32.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:32.539+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:32.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:32.539+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:32.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:32.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:32.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:32.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:32.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:32.391+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:32.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:32.392+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:32.749+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.750341+0000) 2024-02-20T09:27:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:32.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:32.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:32.662+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:32.662+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:33.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:33.021+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65456 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:33.354 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:33.017+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.017942+0000) 2024-02-20T09:27:33.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:33.017+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.018134+0000) 2024-02-20T09:27:33.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:33.046+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.047691+0000) 2024-02-20T09:27:33.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:33.019+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.020136+0000) 2024-02-20T09:27:33.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:33.038+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.039230+0000) 2024-02-20T09:27:33.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:33.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:33.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:33.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:33.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:33.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:33.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:33.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:33.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:33.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:33.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:33.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:33.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:33.507+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:33.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:33.508+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:33.749+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.750703+0000) 2024-02-20T09:27:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:33.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:33.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:33.711+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:33.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:33.712+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:34.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:34.049+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:34.050182+0000) 2024-02-20T09:27:34.343 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:34.041+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:34.042764+0000) 2024-02-20T09:27:34.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:34.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:34.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:34.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:34.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:34.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:34.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:34.342+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:34.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:34.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:34.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:34.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:34.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:34.551+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:34.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:34.551+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:34.752+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:34.753400+0000) 2024-02-20T09:27:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:34.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:34.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:34.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:34.698+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:34.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:34.699+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:35.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:35.051+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:35.052593+0000) 2024-02-20T09:27:35.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:35.044+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:35.045112+0000) 2024-02-20T09:27:35.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:35.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:35.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:35.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:35.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:35.339+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:35.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:35.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:35.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:35.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:35.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:35.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:35.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:35.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:35.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:35.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:35.754+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:35.755908+0000) 2024-02-20T09:27:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:35.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:35.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:35.658+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:35.659+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:36.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:36.053+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:36.054490+0000) 2024-02-20T09:27:36.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:36.046+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:36.047789+0000) 2024-02-20T09:27:36.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:36.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:36.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:36.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:36.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:36.338+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:36.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:36.339+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:36.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:36.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:36.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:36.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:36.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:36.625+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:36.895 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:36.625+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:36.895 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:36.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:36.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:36.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:36.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:36.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:36.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:36.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:36.896 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:36.757+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:36.758393+0000) 2024-02-20T09:27:37.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:37.056+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.057023+0000) 2024-02-20T09:27:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:37.049+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.050295+0000) 2024-02-20T09:27:37.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:37.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:37.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:37.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:37.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:37.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:37.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:37.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:37.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:37.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:37.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:37.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:37.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:37.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:37.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:37.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:37.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:37.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:37.759+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.760844+0000) 2024-02-20T09:27:38.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:38.029+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65495 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:38.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:38.027+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.028124+0000) 2024-02-20T09:27:38.305 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:38.057+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.057955+0000) 2024-02-20T09:27:38.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:38.026+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.027307+0000) 2024-02-20T09:27:38.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:38.049+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.050815+0000) 2024-02-20T09:27:38.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:38.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:38.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:38.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:38.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:38.302+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:38.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:38.024+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.025901+0000) 2024-02-20T09:27:38.582 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:38.303+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:38.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:38.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:38.658+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:38.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:38.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:38.760+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.761329+0000) 2024-02-20T09:27:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:38.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:38.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:38.580+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:38.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:38.581+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:39.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:39.060+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:39.061475+0000) 2024-02-20T09:27:39.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:39.052+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:39.053411+0000) 2024-02-20T09:27:39.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:39.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:39.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:39.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:39.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:39.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:39.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:39.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:39.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:39.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:39.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:39.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:39.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:39.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:39.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:39.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:39.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:39.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:39.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:39.693+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:39.694+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:39.762+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:39.763518+0000) 2024-02-20T09:27:40.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:40.064+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:40.065028+0000) 2024-02-20T09:27:40.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:40.056+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:40.057038+0000) 2024-02-20T09:27:40.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:40.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:40.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:40.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:40.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:40.331+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:40.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:40.533+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:40.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:40.533+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:40.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:40.533+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:40.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:40.534+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:40.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:40.332+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:40.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:40.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:40.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:40.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:40.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:40.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:40.765+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:40.766018+0000) 2024-02-20T09:27:41.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:41.067+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:41.068570+0000) 2024-02-20T09:27:41.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:41.058+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:41.059807+0000) 2024-02-20T09:27:41.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:41.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:41.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:41.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:41.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:41.583+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:41.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:41.583+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:41.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:41.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:41.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:41.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:41.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:41.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:41.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:41.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:41.714+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:41.714+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:41.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:41.714+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:41.715+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:41.767+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:41.768493+0000) 2024-02-20T09:27:42.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:42.070+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.071034+0000) 2024-02-20T09:27:42.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:42.061+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.062333+0000) 2024-02-20T09:27:42.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:42.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:42.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:42.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:42.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:42.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:42.624 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:42.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:42.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:42.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:42.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:42.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:42.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:42.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:42.667+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:42.669+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:42.769+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.770869+0000) 2024-02-20T09:27:43.354 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:43.037+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65535 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:43.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:43.034+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.035439+0000) 2024-02-20T09:27:43.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:43.070+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.071684+0000) 2024-02-20T09:27:43.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:43.032+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.033752+0000) 2024-02-20T09:27:43.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:43.061+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.062749+0000) 2024-02-20T09:27:43.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:43.034+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.035218+0000) 2024-02-20T09:27:43.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:43.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:43.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:43.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:43.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:43.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:43.859 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:43.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:43.859 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:43.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:43.859 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:43.605+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:43.860 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:43.607+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:43.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:43.641+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:43.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:43.641+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:43.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:43.641+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:43.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:43.642+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:43.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:43.770+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.771423+0000) 2024-02-20T09:27:44.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:44.074+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:44.075094+0000) 2024-02-20T09:27:44.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:44.064+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:44.065178+0000) 2024-02-20T09:27:44.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:44.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:44.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:44.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:44.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:44.323+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:44.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:44.324+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:44.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:44.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:44.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:44.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:44.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:44.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:44.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:44.605+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:44.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:44.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:44.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:44.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:44.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:44.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:44.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:44.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:44.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:44.773+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:44.773955+0000) 2024-02-20T09:27:45.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:45.076+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:45.077438+0000) 2024-02-20T09:27:45.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:45.066+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:45.067866+0000) 2024-02-20T09:27:45.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:45.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:45.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:45.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:45.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:45.555+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:45.642 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:45.556+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:45.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:45.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:45.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:45.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:45.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:45.354+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:45.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:45.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:45.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:45.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:45.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:45.639+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:45.640+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:45.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:45.775+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:45.776414+0000) 2024-02-20T09:27:46.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:46.079+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:46.079951+0000) 2024-02-20T09:27:46.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:46.069+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:46.070420+0000) 2024-02-20T09:27:46.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:46.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:46.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:46.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:46.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:46.585+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:46.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:46.585+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:46.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:46.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:46.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:46.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:46.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:46.370+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:46.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:46.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:46.686+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:46.686+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:46.686+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:46.687+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:46.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:46.779+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:46.780061+0000) 2024-02-20T09:27:47.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:47.081+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.082030+0000) 2024-02-20T09:27:47.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:47.071+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.072547+0000) 2024-02-20T09:27:47.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:47.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:47.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:47.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:47.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:47.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:47.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:47.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:47.663 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:47.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:47.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:47.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:47.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:47.406+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:47.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:47.407+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:47.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:47.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:47.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:47.661+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:47.662+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:47.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:47.781+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.782535+0000) 2024-02-20T09:27:48.354 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:48.045+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65574 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:48.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:48.042+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.043315+0000) 2024-02-20T09:27:48.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:48.081+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.082594+0000) 2024-02-20T09:27:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:48.042+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.042947+0000) 2024-02-20T09:27:48.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:48.043+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.044736+0000) 2024-02-20T09:27:48.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:48.072+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.072974+0000) 2024-02-20T09:27:48.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:48.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:48.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:48.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:48.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:48.595+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:48.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:48.596+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:48.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:48.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:48.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:48.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:48.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:48.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:48.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:48.359+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:48.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:48.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:48.654+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:48.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:48.655+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:48.782+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.783017+0000) 2024-02-20T09:27:49.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:49.084+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:49.084975+0000) 2024-02-20T09:27:49.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:49.074+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:49.075861+0000) 2024-02-20T09:27:49.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:49.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:49.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:49.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:49.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:49.551+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:49.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:49.552+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:49.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:49.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:49.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:49.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:49.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:49.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:49.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:49.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:49.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:49.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:49.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:49.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:49.698+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:49.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:49.784+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:49.785433+0000) 2024-02-20T09:27:50.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:50.087+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:50.088117+0000) 2024-02-20T09:27:50.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:50.077+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:50.078298+0000) 2024-02-20T09:27:50.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:50.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:50.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:50.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:50.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:50.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:50.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:50.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:50.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:50.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:50.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:50.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:50.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:50.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:50.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:50.576+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:50.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:50.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:50.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:50.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:50.653+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:50.787+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:50.788504+0000) 2024-02-20T09:27:51.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:51.090+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:51.091222+0000) 2024-02-20T09:27:51.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:51.079+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:51.080704+0000) 2024-02-20T09:27:51.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:51.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:51.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:51.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:51.355 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:51.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:51.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:51.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:51.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:51.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:51.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:51.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:51.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:51.543+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:51.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:51.544+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:51.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:51.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:51.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:51.690+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:51.691+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:51.790+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:51.790953+0000) 2024-02-20T09:27:52.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:52.093+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.094529+0000) 2024-02-20T09:27:52.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:52.082+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.083301+0000) 2024-02-20T09:27:52.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:52.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:52.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:52.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:52.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:52.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:52.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:52.356+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:52.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:52.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:52.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:52.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:52.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:52.578+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:52.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:52.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:52.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:52.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:52.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:52.792+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.793643+0000) 2024-02-20T09:27:53.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:53.052+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.053468+0000) 2024-02-20T09:27:53.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:53.094+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.094931+0000) 2024-02-20T09:27:53.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:53.050+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.051566+0000) 2024-02-20T09:27:53.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:53.082+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.083849+0000) 2024-02-20T09:27:53.347 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:53.054+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65614 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:53.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:53.050+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.051249+0000) 2024-02-20T09:27:53.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:53.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:53.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:53.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:53.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:53.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:53.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:53.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:53.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:53.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:53.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:53.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:53.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:53.344+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:53.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:53.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:53.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:53.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:53.747+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:53.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:53.793+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.794125+0000) 2024-02-20T09:27:54.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:54.096+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:54.097187+0000) 2024-02-20T09:27:54.350 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:54.085+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:54.086264+0000) 2024-02-20T09:27:54.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:54.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:54.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:54.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:54.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:54.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:54.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:54.505+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:54.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:54.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:54.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:54.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:54.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:54.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:54.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:54.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:54.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:54.718+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:54.974 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:54.718+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:54.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:54.718+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:54.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:54.719+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:54.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:54.796+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:54.797629+0000) 2024-02-20T09:27:55.101 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:55.088+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:55.088969+0000) 2024-02-20T09:27:55.106 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:55.099+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:55.100070+0000) 2024-02-20T09:27:55.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:55.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:55.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:55.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:55.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:55.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:55.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:55.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:55.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:55.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:55.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:55.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:55.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:55.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:55.680 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:55.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:55.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:55.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:55.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:55.676+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:55.676+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:55.800+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:55.800931+0000) 2024-02-20T09:27:56.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:56.102+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:56.103058+0000) 2024-02-20T09:27:56.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:56.090+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:56.091582+0000) 2024-02-20T09:27:56.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:56.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:56.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:56.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:56.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:56.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:56.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:56.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:56.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:56.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:56.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:56.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:56.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:56.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:56.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:56.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:56.704+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:56.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:56.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:56.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:56.705+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:56.802+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:56.803632+0000) 2024-02-20T09:27:57.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:57.105+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.106711+0000) 2024-02-20T09:27:57.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:57.093+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.094182+0000) 2024-02-20T09:27:57.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:57.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:57.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:57.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:57.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:57.341+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:57.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:57.342+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:57.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:57.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:57.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:57.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:57.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:57.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:57.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:57.519+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:57.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:57.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:57.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:57.688+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:57.806+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.807187+0000) 2024-02-20T09:27:58.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:58.060+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.061503+0000) 2024-02-20T09:27:58.316 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:58.106+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.107207+0000) 2024-02-20T09:27:58.317 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:27:58.062+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65653 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:27:58.318 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:58.062+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.063001+0000) 2024-02-20T09:27:58.318 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:58.093+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.094684+0000) 2024-02-20T09:27:58.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:58.058+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.059819+0000) 2024-02-20T09:27:58.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:58.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:58.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:58.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:58.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:58.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:58.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:58.554+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:58.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:58.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:58.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:58.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:58.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:58.314+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:58.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:58.315+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:58.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:58.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:58.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:58.652+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:58.653+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:58.806+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.807785+0000) 2024-02-20T09:27:59.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:59.108+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:59.109824+0000) 2024-02-20T09:27:59.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:59.096+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:59.097356+0000) 2024-02-20T09:27:59.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:59.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:59.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:59.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:59.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:59.274+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:27:59.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:27:59.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:59.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:59.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:59.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:59.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:59.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:59.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:27:59.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:27:59.577+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:59.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:59.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:59.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:27:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:59.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:27:59.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:27:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:27:59.809+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:59.810712+0000) 2024-02-20T09:28:00.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:00.111+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:00.112083+0000) 2024-02-20T09:28:00.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:00.099+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:00.099949+0000) 2024-02-20T09:28:00.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:00.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:00.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:00.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:00.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:00.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:00.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:00.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:00.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:00.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:00.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:00.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:00.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:00.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:00.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:00.589+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:00.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:00.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:00.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:00.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:00.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:00.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:00.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:00.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:00.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:00.812+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:00.813148+0000) 2024-02-20T09:28:01.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:01.113+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:01.114533+0000) 2024-02-20T09:28:01.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:01.101+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:01.102162+0000) 2024-02-20T09:28:01.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:01.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:01.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:01.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:01.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:01.252+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:01.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:01.253+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:01.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:01.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:01.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:01.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:01.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:01.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:01.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:01.585+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:01.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:01.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:01.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:01.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:01.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:01.594+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:01.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:01.595+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:01.814+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:01.815745+0000) 2024-02-20T09:28:02.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:02.116+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.117034+0000) 2024-02-20T09:28:02.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:02.103+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.104573+0000) 2024-02-20T09:28:02.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:02.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:02.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:02.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:02.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:02.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:02.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:02.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:02.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:02.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:02.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:02.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:02.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:02.562+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:02.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:02.563+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:02.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:02.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:02.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:02.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:02.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:02.561+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:02.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:02.562+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:03.069 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:03.067+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.068424+0000) 2024-02-20T09:28:03.070 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:02.817+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.818330+0000) 2024-02-20T09:28:03.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:03.067+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.068835+0000) 2024-02-20T09:28:03.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:03.116+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.117725+0000) 2024-02-20T09:28:03.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:03.104+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.105375+0000) 2024-02-20T09:28:03.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:03.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:03.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:03.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:03.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:03.295+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:03.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:03.296+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:03.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:03.070+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65693 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:03.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:03.068+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.069120+0000) 2024-02-20T09:28:03.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:03.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:03.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:03.576+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:03.577+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:03.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:03.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:03.582+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:03.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:03.583+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:03.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:03.817+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.818974+0000) 2024-02-20T09:28:04.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:04.118+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:04.120156+0000) 2024-02-20T09:28:04.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:04.105+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:04.107842+0000) 2024-02-20T09:28:04.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:04.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:04.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:04.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:04.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:04.272+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:04.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:04.274+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:04.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:04.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:04.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:04.622+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:04.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:04.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:04.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:04.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:04.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:04.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:05.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:04.819+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:04.821115+0000) 2024-02-20T09:28:05.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:05.120+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:05.122536+0000) 2024-02-20T09:28:05.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:05.108+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:05.110359+0000) 2024-02-20T09:28:05.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:05.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:05.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:05.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:05.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:05.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:05.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:05.290+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:05.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:05.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:05.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:05.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:05.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:05.653+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:05.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:05.654+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:05.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:05.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:05.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:05.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:05.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:05.585+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:05.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:05.585+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:05.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:05.821+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:05.823745+0000) 2024-02-20T09:28:06.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:06.123+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:06.125419+0000) 2024-02-20T09:28:06.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:06.111+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:06.112976+0000) 2024-02-20T09:28:06.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:06.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:06.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:06.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:06.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:06.277+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:06.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:06.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:06.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:06.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:06.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:06.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:06.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:06.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:06.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:06.601+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:06.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:06.825+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:06.826274+0000) 2024-02-20T09:28:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:06.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:06.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:06.697+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:06.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:06.698+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:07.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:07.125+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.127815+0000) 2024-02-20T09:28:07.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:07.113+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.115120+0000) 2024-02-20T09:28:07.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:07.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:07.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:07.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:07.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:07.276+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:07.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:07.278+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:07.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:07.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:07.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:07.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:07.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:07.650+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:07.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:07.651+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:07.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:07.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:07.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:07.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:07.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:07.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:07.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:07.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:07.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:07.828+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.828968+0000) 2024-02-20T09:28:08.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:08.075+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.076334+0000) 2024-02-20T09:28:08.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:08.127+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.128279+0000) 2024-02-20T09:28:08.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:08.076+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.077588+0000) 2024-02-20T09:28:08.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:08.114+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.115553+0000) 2024-02-20T09:28:08.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:08.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:08.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:08.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:08.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:08.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:08.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:08.262+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:08.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:08.078+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65732 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:08.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:08.075+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.076473+0000) 2024-02-20T09:28:08.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:08.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:08.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:08.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:08.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:08.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:08.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:08.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:08.699+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:08.699+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:08.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:08.828+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.829524+0000) 2024-02-20T09:28:09.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:09.130+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:09.131625+0000) 2024-02-20T09:28:09.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:09.117+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:09.118016+0000) 2024-02-20T09:28:09.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:09.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:09.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:09.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:09.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:09.289+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:09.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:09.290+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:09.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:09.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:09.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:09.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:09.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:09.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:09.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:09.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:09.831+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:09.831957+0000) 2024-02-20T09:28:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:09.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:09.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:09.690+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:09.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:09.691+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:10.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:10.133+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:10.134246+0000) 2024-02-20T09:28:10.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:10.119+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:10.120272+0000) 2024-02-20T09:28:10.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:10.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:10.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:10.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:10.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:10.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:10.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:10.272+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:10.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:10.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:10.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:10.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:10.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:10.676+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:10.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:10.677+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:10.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:10.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:10.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:10.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:10.833+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:10.834625+0000) 2024-02-20T09:28:11.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:11.135+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:11.136923+0000) 2024-02-20T09:28:11.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:11.122+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:11.122929+0000) 2024-02-20T09:28:11.429 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:11.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:11.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:11.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:11.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:11.310+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:11.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:11.312+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:11.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:11.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:11.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:11.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:11.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:11.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:11.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:11.650+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:11.982 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:11.836+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:11.837164+0000) 2024-02-20T09:28:11.982 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:11.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:11.982 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:11.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:11.982 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:11.724+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:11.982 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:11.726+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:12.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:12.138+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.139373+0000) 2024-02-20T09:28:12.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:12.124+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.125412+0000) 2024-02-20T09:28:12.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:12.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:12.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:12.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:12.348 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:12.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:12.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:12.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:12.677+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:12.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:12.678+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:12.679+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:12.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:12.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:12.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:12.663+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:12.838+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.839240+0000) 2024-02-20T09:28:13.357 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:13.087+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65772 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:13.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:13.083+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.084741+0000) 2024-02-20T09:28:13.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:13.138+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.139813+0000) 2024-02-20T09:28:13.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:13.085+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.086779+0000) 2024-02-20T09:28:13.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:13.125+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.126074+0000) 2024-02-20T09:28:13.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:13.083+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.084513+0000) 2024-02-20T09:28:13.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:13.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:13.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:13.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:13.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:13.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:13.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:13.380+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:13.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:13.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:13.648+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:13.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:13.648+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:13.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:13.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:13.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:13.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:13.838+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.839683+0000) 2024-02-20T09:28:14.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:14.142+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:14.143333+0000) 2024-02-20T09:28:14.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:14.128+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:14.129208+0000) 2024-02-20T09:28:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:14.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:14.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:14.397+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:14.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:14.398+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:14.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:14.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:14.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:14.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:14.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:14.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:14.841+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:14.842336+0000) 2024-02-20T09:28:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:14.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:14.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:14.660+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:14.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:14.661+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:15.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:15.144+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:15.145768+0000) 2024-02-20T09:28:15.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:15.131+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:15.132554+0000) 2024-02-20T09:28:15.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:15.649+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:15.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:15.649+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:15.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:15.649+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:15.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:15.650+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:15.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:15.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:15.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:15.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:15.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:15.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:15.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:15.630+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:15.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:15.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:15.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:15.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:15.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:15.844+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:15.845785+0000) 2024-02-20T09:28:16.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:16.148+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:16.149304+0000) 2024-02-20T09:28:16.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:16.133+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:16.134626+0000) 2024-02-20T09:28:16.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:16.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:16.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:16.619+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:16.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:16.620+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:16.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:16.620+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:16.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:16.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:16.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:16.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:16.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:16.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:16.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:16.437+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:16.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:16.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:16.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:16.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:16.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:16.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:16.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:16.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:16.847+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:16.848351+0000) 2024-02-20T09:28:17.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:17.151+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.152875+0000) 2024-02-20T09:28:17.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:17.137+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.138275+0000) 2024-02-20T09:28:17.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:17.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:17.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:17.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:17.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:17.631+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:17.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:17.632+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:17.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:17.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:17.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:17.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:17.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:17.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:17.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:17.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:17.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:17.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:17.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:17.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:17.436+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:17.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:17.437+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:17.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:17.850+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.850953+0000) 2024-02-20T09:28:18.357 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:18.095+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65811 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:18.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:18.092+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.093324+0000) 2024-02-20T09:28:18.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:18.091+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.092857+0000) 2024-02-20T09:28:18.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:18.152+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.153364+0000) 2024-02-20T09:28:18.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:18.091+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.092738+0000) 2024-02-20T09:28:18.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:18.138+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.138951+0000) 2024-02-20T09:28:18.646 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:18.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:18.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:18.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:18.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:18.602+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:18.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:18.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:18.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:18.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:18.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:18.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:18.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:18.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:18.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:18.477+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:18.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:18.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:18.644+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:18.645+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:18.850+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.851401+0000) 2024-02-20T09:28:19.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:19.154+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:19.155463+0000) 2024-02-20T09:28:19.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:19.140+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:19.141660+0000) 2024-02-20T09:28:19.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:19.571+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:19.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:19.571+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:19.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:19.571+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:19.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:19.572+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:19.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:19.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:19.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:19.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:19.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:19.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:19.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:19.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:19.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:19.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:19.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:19.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:19.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:19.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:19.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:19.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:19.852+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:19.853628+0000) 2024-02-20T09:28:20.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:20.156+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:20.157494+0000) 2024-02-20T09:28:20.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:20.143+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:20.143949+0000) 2024-02-20T09:28:20.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:20.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:20.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:20.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:20.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:20.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:20.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:20.545+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:20.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:20.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:20.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:20.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:20.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:20.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:20.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:20.616+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:20.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:20.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:20.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:20.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:20.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:20.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:20.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:20.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:20.855+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:20.856191+0000) 2024-02-20T09:28:21.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:21.159+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:21.160857+0000) 2024-02-20T09:28:21.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:21.145+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:21.146370+0000) 2024-02-20T09:28:21.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:21.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:21.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:21.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:21.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:21.497+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:21.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:21.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:21.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:21.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:21.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:21.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:21.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:21.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:21.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:21.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:21.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:21.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:21.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:21.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:21.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:21.642+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:21.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:21.643+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:21.857+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:21.858298+0000) 2024-02-20T09:28:22.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:22.163+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.164174+0000) 2024-02-20T09:28:22.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:22.149+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.149969+0000) 2024-02-20T09:28:22.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:22.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:22.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:22.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:22.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:22.514+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:22.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:22.515+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:22.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:22.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:22.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:22.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:22.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:22.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:22.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:22.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:22.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:22.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:22.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:22.632+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:22.859+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.860799+0000) 2024-02-20T09:28:23.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:23.103+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65851 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:23.358 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:23.099+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.100862+0000) 2024-02-20T09:28:23.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:23.101+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.102322+0000) 2024-02-20T09:28:23.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:23.163+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.164752+0000) 2024-02-20T09:28:23.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:23.100+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.101240+0000) 2024-02-20T09:28:23.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:23.149+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.150556+0000) 2024-02-20T09:28:23.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:23.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:23.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:23.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:23.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:23.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:23.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:23.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:23.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:23.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:23.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:23.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:23.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:23.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:23.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:23.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:23.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:23.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:23.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:23.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:23.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:23.860+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.861398+0000) 2024-02-20T09:28:24.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:24.167+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:24.167953+0000) 2024-02-20T09:28:24.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:24.152+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:24.153788+0000) 2024-02-20T09:28:24.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:24.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:24.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:24.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:24.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:24.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:24.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:24.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:24.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:24.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:24.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:24.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:24.500+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:24.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:24.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:24.613+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:24.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:24.614+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:24.862+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:24.863679+0000) 2024-02-20T09:28:25.423 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:25.156+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:25.157342+0000) 2024-02-20T09:28:25.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:25.170+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:25.171309+0000) 2024-02-20T09:28:25.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:25.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:25.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:25.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:25.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:25.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:25.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:25.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:25.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:25.461+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:25.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:25.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:25.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:25.462+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:25.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:25.463+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:25.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:25.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:25.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:25.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:25.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:25.593+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:25.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:25.594+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:25.865+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:25.866169+0000) 2024-02-20T09:28:26.358 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:26.173+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:26.174748+0000) 2024-02-20T09:28:26.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:26.159+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:26.160352+0000) 2024-02-20T09:28:26.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:26.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:26.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:26.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:26.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:26.416+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:26.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:26.418+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:26.617 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:26.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:26.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:26.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:26.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:26.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:26.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:26.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:26.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:26.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:26.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:26.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:26.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:26.615+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:26.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:26.616+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:27.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:26.867+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:26.868396+0000) 2024-02-20T09:28:27.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:27.175+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.176909+0000) 2024-02-20T09:28:27.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:27.161+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.162645+0000) 2024-02-20T09:28:27.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:27.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:27.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:27.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:27.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:27.441+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:27.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:27.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:27.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:27.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:27.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:27.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:27.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:27.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:27.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:27.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:27.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:27.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:27.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:27.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:27.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:27.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:27.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:27.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:27.871+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.871953+0000) 2024-02-20T09:28:28.358 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:28.111+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65890 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:28.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:28.108+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.109677+0000) 2024-02-20T09:28:28.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:28.109+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.110405+0000) 2024-02-20T09:28:28.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:28.176+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.177418+0000) 2024-02-20T09:28:28.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:28.108+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.109653+0000) 2024-02-20T09:28:28.360 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:28.162+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.163416+0000) 2024-02-20T09:28:28.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:28.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:28.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:28.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:28.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:28.508+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:28.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:28.509+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:28.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:28.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:28.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:28.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:28.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:28.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:28.613 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:28.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:28.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:28.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:28.873 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:28.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:28.874 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:28.610+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:28.874 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:28.610+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:29.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:28.871+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.872443+0000) 2024-02-20T09:28:29.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:29.180+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:29.180955+0000) 2024-02-20T09:28:29.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:29.164+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:29.165526+0000) 2024-02-20T09:28:29.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:29.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:29.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:29.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:29.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:29.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:29.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:29.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:29.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:29.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:29.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:29.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:29.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:29.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:29.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:29.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:29.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:29.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:29.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:29.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:29.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:29.541+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:29.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:29.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:29.873+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:29.874693+0000) 2024-02-20T09:28:30.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:30.182+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:30.183293+0000) 2024-02-20T09:28:30.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:30.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:30.167620+0000) 2024-02-20T09:28:30.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:30.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:30.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:30.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:30.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:30.550+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:30.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:30.551+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:30.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:30.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:30.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:30.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:30.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:30.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:30.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:30.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:30.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:30.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:30.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:30.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:30.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:30.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:30.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:30.628+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:30.875+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:30.876863+0000) 2024-02-20T09:28:31.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:31.184+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:31.185326+0000) 2024-02-20T09:28:31.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:31.170+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:31.171217+0000) 2024-02-20T09:28:31.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:31.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:31.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:31.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:31.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:31.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:31.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:31.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:31.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:31.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:31.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:31.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:31.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:31.577+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:31.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:31.578+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:31.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:31.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:31.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:31.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:31.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:31.643+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:31.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:31.644+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:31.878+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:31.879505+0000) 2024-02-20T09:28:32.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:32.187+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.188780+0000) 2024-02-20T09:28:32.359 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:32.173+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.174788+0000) 2024-02-20T09:28:32.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:32.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:32.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:32.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:32.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:32.409+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:32.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:32.410+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:32.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:32.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:32.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:32.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:32.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:32.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:32.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:32.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:32.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:32.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:32.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:32.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:32.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:32.622+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:32.884 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:32.623+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:33.135 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:33.119+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65930 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:33.135 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:33.117+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.118680+0000) 2024-02-20T09:28:33.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:33.118+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.119202+0000) 2024-02-20T09:28:33.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:32.882+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.883026+0000) 2024-02-20T09:28:33.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:33.116+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.117776+0000) 2024-02-20T09:28:33.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:33.174+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.175305+0000) 2024-02-20T09:28:33.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:33.188+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.189291+0000) 2024-02-20T09:28:33.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:33.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:33.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:33.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:33.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:33.373+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:33.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:33.374+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:33.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:33.608+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:33.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:33.608+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:33.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:33.608+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:33.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:33.609+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:33.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:33.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:33.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:33.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:33.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:33.630+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:33.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:33.631+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:33.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:33.882+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.883776+0000) 2024-02-20T09:28:34.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:34.176+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:34.177680+0000) 2024-02-20T09:28:34.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:34.190+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:34.191809+0000) 2024-02-20T09:28:34.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:34.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:34.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:34.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:34.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:34.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:34.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:34.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:34.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:34.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:34.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:34.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:34.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:34.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:34.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:34.665+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:34.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:34.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:34.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:34.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:34.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:34.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:34.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:34.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:34.885+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:34.886350+0000) 2024-02-20T09:28:35.431 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:35.178+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:35.179840+0000) 2024-02-20T09:28:35.431 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:35.194+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:35.195099+0000) 2024-02-20T09:28:35.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:35.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:35.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:35.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:35.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:35.360+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:35.432 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:35.361+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:35.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:35.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:35.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:35.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:35.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:35.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:35.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:35.543+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:35.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:35.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:35.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:35.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:35.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:35.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:35.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:35.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:35.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:35.888+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:35.888933+0000) 2024-02-20T09:28:36.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:36.182+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:36.183252+0000) 2024-02-20T09:28:36.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:36.197+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:36.198446+0000) 2024-02-20T09:28:36.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:36.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:36.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:36.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:36.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:36.395+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:36.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:36.396+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:36.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:36.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:36.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:36.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:36.514+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:36.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:36.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:36.601+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:36.602+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:37.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:36.890+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:36.891417+0000) 2024-02-20T09:28:37.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:37.184+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.185245+0000) 2024-02-20T09:28:37.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:37.201+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.202039+0000) 2024-02-20T09:28:37.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:37.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:37.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:37.396+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:37.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:37.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:37.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:37.397+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:37.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:37.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:37.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:37.564+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:37.564+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:37.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:37.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:37.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:37.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:38.149 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:38.127+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 65969 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:38.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:38.125+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.126392+0000) 2024-02-20T09:28:38.150 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:38.123+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.124686+0000) 2024-02-20T09:28:38.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:37.892+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.893650+0000) 2024-02-20T09:28:38.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:38.123+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.124594+0000) 2024-02-20T09:28:38.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:38.201+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.202798+0000) 2024-02-20T09:28:38.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:38.184+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.185775+0000) 2024-02-20T09:28:38.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:38.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:38.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:38.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:38.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:38.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:38.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:38.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:38.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:38.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:38.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:38.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:38.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:38.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:38.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:38.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:38.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:38.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:38.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:38.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:38.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:38.645+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:38.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:38.646+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:38.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:38.893+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.894246+0000) 2024-02-20T09:28:39.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:39.204+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:39.205012+0000) 2024-02-20T09:28:39.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:39.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:39.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:39.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:39.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:39.384+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:39.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:39.385+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:39.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:39.187+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:39.188869+0000) 2024-02-20T09:28:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:39.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:39.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:39.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:39.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:39.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:39.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:39.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:39.618+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:39.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:39.619+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:40.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:39.896+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:39.897333+0000) 2024-02-20T09:28:40.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:40.207+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:40.208555+0000) 2024-02-20T09:28:40.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:40.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:40.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:40.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:40.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:40.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:40.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:40.381+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:40.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:40.190+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:40.191125+0000) 2024-02-20T09:28:40.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:40.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:40.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:40.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:40.597+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:40.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:40.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:40.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:40.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:40.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:40.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:41.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:40.898+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:40.899860+0000) 2024-02-20T09:28:41.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:41.192+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:41.193671+0000) 2024-02-20T09:28:41.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:41.210+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:41.211108+0000) 2024-02-20T09:28:41.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:41.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:41.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:41.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:41.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:41.427+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:41.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:41.428+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:41.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:41.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:41.635+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:41.636+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:41.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:41.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:41.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:41.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:41.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:42.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:41.901+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:41.902353+0000) 2024-02-20T09:28:42.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:42.196+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.197044+0000) 2024-02-20T09:28:42.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:42.212+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.213628+0000) 2024-02-20T09:28:42.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:42.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:42.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:42.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:42.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:42.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:42.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:42.442+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:42.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:42.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:42.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:42.640+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:42.640+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:42.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:42.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:42.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:42.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:42.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:43.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:43.136+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66009 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:43.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:43.131+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.132741+0000) 2024-02-20T09:28:43.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:43.133+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.134564+0000) 2024-02-20T09:28:43.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:42.905+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.905950+0000) 2024-02-20T09:28:43.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:43.132+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.133181+0000) 2024-02-20T09:28:43.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:43.213+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.214210+0000) 2024-02-20T09:28:43.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:43.196+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.197678+0000) 2024-02-20T09:28:43.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:43.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:43.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:43.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:43.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:43.473+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:43.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:43.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:43.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:43.626+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:43.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:43.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:43.616+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:43.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:43.616+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:44.201 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:43.905+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.906603+0000) 2024-02-20T09:28:44.463 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:44.216+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:44.217442+0000) 2024-02-20T09:28:44.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:44.200+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:44.200957+0000) 2024-02-20T09:28:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:44.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:44.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:44.489+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:44.490+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:44.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:44.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:44.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:44.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:44.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:44.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:44.591+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:44.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:44.592+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:45.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:44.908+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:44.909258+0000) 2024-02-20T09:28:45.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:45.219+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:45.220767+0000) 2024-02-20T09:28:45.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:45.203+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:45.204299+0000) 2024-02-20T09:28:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:45.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:45.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:45.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:45.510+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:45.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:45.511+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:45.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:45.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:45.591+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:45.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:45.592+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:45.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:45.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:45.586+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:45.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:45.587+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:46.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:45.911+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:45.912542+0000) 2024-02-20T09:28:46.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:46.221+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:46.222877+0000) 2024-02-20T09:28:46.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:46.205+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:46.206852+0000) 2024-02-20T09:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:46.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:46.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:46.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:46.537+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:46.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:46.538+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:46.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:46.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:46.622+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:46.623+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:46.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:46.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:46.578+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:46.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:46.579+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:47.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:46.913+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:46.914582+0000) 2024-02-20T09:28:47.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:47.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.226394+0000) 2024-02-20T09:28:47.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:47.208+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.209330+0000) 2024-02-20T09:28:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:47.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:47.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:47.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:47.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:47.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:47.557+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:47.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:47.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:47.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:47.586+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:47.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:47.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:47.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:47.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:47.617+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:48.207 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:48.144+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66048 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:48.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:48.140+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.141347+0000) 2024-02-20T09:28:48.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:48.141+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.142734+0000) 2024-02-20T09:28:48.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:47.916+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.917153+0000) 2024-02-20T09:28:48.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:48.140+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.141841+0000) 2024-02-20T09:28:48.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:48.208+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.209852+0000) 2024-02-20T09:28:48.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:48.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.226925+0000) 2024-02-20T09:28:48.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:48.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:48.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:48.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:48.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:48.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:48.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:48.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:48.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:48.574+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:48.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:48.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:48.649+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:48.649+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:49.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:48.916+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.917663+0000) 2024-02-20T09:28:49.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:49.229+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:49.230212+0000) 2024-02-20T09:28:49.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:49.211+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:49.212267+0000) 2024-02-20T09:28:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:49.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:49.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:49.480+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:49.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:49.481+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:49.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:49.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:49.598+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:49.599+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:49.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:49.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:49.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:49.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:50.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:49.918+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:49.919626+0000) 2024-02-20T09:28:50.464 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:50.213+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:50.214317+0000) 2024-02-20T09:28:50.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:50.231+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:50.232871+0000) 2024-02-20T09:28:50.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:50.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:50.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:50.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:50.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:50.450+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:50.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:50.451+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:50.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:50.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:50.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:50.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:50.683+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:50.683+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:50.683+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:50.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:50.684+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:51.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:50.921+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:50.922035+0000) 2024-02-20T09:28:51.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:51.234+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:51.235335+0000) 2024-02-20T09:28:51.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:51.216+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:51.216967+0000) 2024-02-20T09:28:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:51.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:51.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:51.488+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:51.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:51.489+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:51.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:51.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:51.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:51.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:51.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:51.610+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:51.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:51.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:51.703+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:51.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:51.704+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:52.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:51.923+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:51.924080+0000) 2024-02-20T09:28:52.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:52.237+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.238665+0000) 2024-02-20T09:28:52.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:52.218+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.219407+0000) 2024-02-20T09:28:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:52.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:52.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:52.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:52.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:52.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:52.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:52.538+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:52.538+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:52.538+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:52.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:52.539+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:53.147 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:52.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:53.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:52.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:53.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:52.746+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:53.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:52.747+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:53.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:52.925+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.926324+0000) 2024-02-20T09:28:53.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:53.148+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.149364+0000) 2024-02-20T09:28:53.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:53.238+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.239301+0000) 2024-02-20T09:28:53.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:53.149+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.150149+0000) 2024-02-20T09:28:53.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:53.219+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.219942+0000) 2024-02-20T09:28:53.434 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:53.152+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66088 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:53.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:53.149+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.150263+0000) 2024-02-20T09:28:53.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:53.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:53.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:53.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:53.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:53.581+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:53.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:53.582+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:53.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:53.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:53.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:53.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:53.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:53.619+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:53.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:53.620+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:53.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:53.726+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:53.726+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:53.726+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:53.727+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:53.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:53.926+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.927118+0000) 2024-02-20T09:28:54.464 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:54.241+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:54.242724+0000) 2024-02-20T09:28:54.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:54.221+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:54.222139+0000) 2024-02-20T09:28:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:54.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:54.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:54.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:54.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:54.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:54.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:54.642+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:54.642+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:54.642+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:54.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:54.643+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:54.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:54.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:54.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:54.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:54.681+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:55.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:54.929+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:54.930594+0000) 2024-02-20T09:28:55.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:55.244+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:55.245263+0000) 2024-02-20T09:28:55.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:55.224+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:55.225419+0000) 2024-02-20T09:28:55.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:55.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:55.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:55.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:55.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:55.575+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:55.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:55.576+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:55.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:55.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:55.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:55.682+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:55.682+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:55.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:55.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:55.677+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:55.936 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:55.678+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:56.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:55.933+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:55.934018+0000) 2024-02-20T09:28:56.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:56.246+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:56.247730+0000) 2024-02-20T09:28:56.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:56.227+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:56.227953+0000) 2024-02-20T09:28:56.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:56.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:56.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:56.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:56.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:56.567+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:56.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:56.568+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:56.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:56.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:56.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:56.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:56.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:56.686+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:56.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:56.687+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:56.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:56.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:56.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:56.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:56.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:56.708+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:56.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:56.709+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:57.217 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:56.934+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:56.936740+0000) 2024-02-20T09:28:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:57.248+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.250787+0000) 2024-02-20T09:28:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:57.229+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.231475+0000) 2024-02-20T09:28:57.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:57.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:57.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:57.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:57.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:57.518+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:57.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:57.519+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:57.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:57.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:57.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:57.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:57.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:57.725+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:57.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:57.726+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:57.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:57.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:57.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:57.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:57.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:57.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:57.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:57.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:58.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:28:58.158+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66127 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:28:58.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:58.155+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.157439+0000) 2024-02-20T09:28:58.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:58.155+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.157431+0000) 2024-02-20T09:28:58.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:57.938+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.940408+0000) 2024-02-20T09:28:58.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:58.155+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.157098+0000) 2024-02-20T09:28:58.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:58.249+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.251406+0000) 2024-02-20T09:28:58.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:58.230+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.232011+0000) 2024-02-20T09:28:58.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:58.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:58.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:58.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:58.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:58.496+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:58.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:58.497+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:58.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:58.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:58.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:58.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:58.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:58.712+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:58.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:58.713+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:58.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:58.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:58.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:58.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:58.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:58.728+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:58.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:58.729+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:59.220 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:58.939+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.941077+0000) 2024-02-20T09:28:59.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:59.232+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:59.234265+0000) 2024-02-20T09:28:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:59.251+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:59.253493+0000) 2024-02-20T09:28:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:59.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:59.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:59.447+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:28:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:28:59.448+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:28:59.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:59.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:59.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:59.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:59.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:59.704+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:28:59.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:28:59.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:59.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:59.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:59.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:59.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:59.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:59.705+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:28:59.946 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:59.706+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:00.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:28:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:28:59.942+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:59.943815+0000) 2024-02-20T09:29:00.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:00.255+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:00.256883+0000) 2024-02-20T09:29:00.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:00.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:00.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:00.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:00.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:00.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:00.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:00.429+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:00.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:00.235+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:00.236941+0000) 2024-02-20T09:29:00.948 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:00.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:00.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:00.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:00.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:00.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:00.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:00.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:00.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:00.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:00.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:00.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:00.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:00.680+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:00.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:00.681+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:01.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:00.946+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:00.947378+0000) 2024-02-20T09:29:01.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:01.238+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:01.239261+0000) 2024-02-20T09:29:01.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:01.258+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:01.259696+0000) 2024-02-20T09:29:01.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:01.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:01.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:01.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:01.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:01.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:01.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:01.419+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:01.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:01.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:01.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:01.693+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:01.694+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:01.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:01.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:01.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:01.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:02.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:01.949+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:01.950371+0000) 2024-02-20T09:29:02.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:02.261+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.262161+0000) 2024-02-20T09:29:02.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:02.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:02.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:02.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:02.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:02.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:02.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:02.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:02.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:02.241+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.242780+0000) 2024-02-20T09:29:02.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:02.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:02.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:02.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:02.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:02.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:02.911 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:02.737+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:02.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:02.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:02.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:02.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:02.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:02.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:02.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:02.628+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:03.162 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:02.951+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.952863+0000) 2024-02-20T09:29:03.434 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:03.168+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66167 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:03.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:03.165+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.166762+0000) 2024-02-20T09:29:03.434 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:03.242+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.243561+0000) 2024-02-20T09:29:03.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:03.164+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.165301+0000) 2024-02-20T09:29:03.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:03.261+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.262704+0000) 2024-02-20T09:29:03.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:03.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:03.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:03.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:03.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:03.359+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:03.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:03.360+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:03.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:03.164+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.165400+0000) 2024-02-20T09:29:03.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:03.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:03.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:03.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:03.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:03.646+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:03.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:03.647+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:03.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:03.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:03.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:03.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:03.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:03.696+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:03.955 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:03.697+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:03.952+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.953646+0000) 2024-02-20T09:29:04.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:04.244+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:04.245702+0000) 2024-02-20T09:29:04.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:04.264+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:04.265084+0000) 2024-02-20T09:29:04.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:04.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:04.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:04.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:04.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:04.382+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:04.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:04.383+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:04.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:04.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:04.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:04.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:04.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:04.729+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:04.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:04.730+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:04.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:04.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:04.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:04.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:04.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:04.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:04.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:04.647+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:05.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:04.954+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:04.955753+0000) 2024-02-20T09:29:05.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:05.248+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:05.249198+0000) 2024-02-20T09:29:05.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:05.267+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:05.268573+0000) 2024-02-20T09:29:05.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:05.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:05.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:05.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:05.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:05.341+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:05.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:05.342+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:05.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:05.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:05.959 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:05.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:05.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:05.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:05.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:05.625+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:05.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:05.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:05.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:05.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:05.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:05.736+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:05.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:05.737+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:06.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:05.957+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:05.958501+0000) 2024-02-20T09:29:06.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:06.250+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:06.251786+0000) 2024-02-20T09:29:06.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:06.270+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:06.271018+0000) 2024-02-20T09:29:06.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:06.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:06.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:06.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:06.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:06.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:06.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:06.341+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:06.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:06.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:06.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:06.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:06.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:06.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:06.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:06.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:06.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:06.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:06.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:06.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:06.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:06.714+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:06.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:06.715+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:07.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:06.961+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:06.962088+0000) 2024-02-20T09:29:07.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:07.254+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.255349+0000) 2024-02-20T09:29:07.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:07.272+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.273558+0000) 2024-02-20T09:29:07.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:07.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:07.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:07.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:07.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:07.368+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:07.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:07.369+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:07.965 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:07.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:07.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:07.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:07.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:07.713+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:07.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:07.714+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:07.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:07.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:07.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:07.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:07.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:07.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:07.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:07.637+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:08.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:08.175+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66206 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:08.172+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.173870+0000) 2024-02-20T09:29:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:08.173+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.174733+0000) 2024-02-20T09:29:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:07.963+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.964569+0000) 2024-02-20T09:29:08.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:08.173+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.173952+0000) 2024-02-20T09:29:08.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:08.255+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.256205+0000) 2024-02-20T09:29:08.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:08.273+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.274381+0000) 2024-02-20T09:29:08.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:08.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:08.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:08.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:08.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:08.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:08.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:08.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:08.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:08.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:08.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:08.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:08.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:08.680+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:08.753 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:08.681+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:09.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:08.964+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.965718+0000) 2024-02-20T09:29:09.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:08.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:08.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:08.750+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:08.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:09.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:09.257+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:09.258816+0000) 2024-02-20T09:29:09.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:09.276+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:09.277718+0000) 2024-02-20T09:29:09.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:09.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:09.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:09.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:09.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:09.372+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:09.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:09.373+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:09.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:09.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:09.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:09.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:09.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:09.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:09.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:09.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:09.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:09.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:09.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:10.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:09.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:10.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:09.968+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:09.969005+0000) 2024-02-20T09:29:10.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:10.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:10.261267+0000) 2024-02-20T09:29:10.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:10.279+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:10.279972+0000) 2024-02-20T09:29:10.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:10.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:10.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:10.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:10.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:10.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:10.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:10.404+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:10.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:10.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:10.731+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:10.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:10.732+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:10.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:10.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:10.701+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:10.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:10.702+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:11.233 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:10.970+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:10.971411+0000) 2024-02-20T09:29:11.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:11.281+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:11.282465+0000) 2024-02-20T09:29:11.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:11.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:11.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:11.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:11.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:11.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:11.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:11.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:11.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:11.263+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:11.264684+0000) 2024-02-20T09:29:12.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:11.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:12.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:11.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:12.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:11.765+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:12.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:11.766+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:11.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:11.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:11.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:11.753+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:12.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:11.973+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:11.974752+0000) 2024-02-20T09:29:12.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:12.285+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.286066+0000) 2024-02-20T09:29:12.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:12.267+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.268310+0000) 2024-02-20T09:29:12.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:12.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:12.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:12.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:12.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:12.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:12.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:12.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:12.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:12.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:12.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:12.815+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:12.720+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:12.720+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:12.720+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:12.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:12.721+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:13.233 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:13.180+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.181350+0000) 2024-02-20T09:29:13.234 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:13.181+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.181963+0000) 2024-02-20T09:29:13.234 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:13.184+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66247 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:13.234 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:12.976+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.977231+0000) 2024-02-20T09:29:13.235 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:13.180+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.181596+0000) 2024-02-20T09:29:13.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:13.286+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.287165+0000) 2024-02-20T09:29:13.495 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:13.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:13.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:13.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:13.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:13.386+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:13.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:13.386+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:13.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:13.268+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.269137+0000) 2024-02-20T09:29:13.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:13.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:13.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:13.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:13.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:13.739+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:13.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:13.740+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:13.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:13.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:13.808+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:13.809+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:14.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:13.976+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.977792+0000) 2024-02-20T09:29:14.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:14.288+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:14.289746+0000) 2024-02-20T09:29:14.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:14.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:14.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:14.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:14.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:14.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:14.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:14.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:14.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:14.270+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:14.271529+0000) 2024-02-20T09:29:15.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:14.827+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:15.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:14.827+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:15.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:14.827+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:15.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:14.828+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:15.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:14.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:15.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:14.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:15.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:14.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:15.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:14.778+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:15.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:14.979+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:14.980276+0000) 2024-02-20T09:29:15.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:15.273+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:15.273953+0000) 2024-02-20T09:29:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:15.291+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:15.292307+0000) 2024-02-20T09:29:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:15.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:15.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:15.397+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:15.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:15.398+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:15.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:16.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:15.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:15.802+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:15.802+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:15.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:15.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:15.774+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:15.775+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:16.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:15.981+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:15.982797+0000) 2024-02-20T09:29:16.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:16.275+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:16.276399+0000) 2024-02-20T09:29:16.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:16.294+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:16.295695+0000) 2024-02-20T09:29:16.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:16.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:16.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:16.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:16.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:16.429+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:16.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:16.430+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:16.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:17.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:16.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:16.824+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:17.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:16.825+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:16.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:16.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:16.765+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:17.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:16.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:17.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:16.984+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:16.985599+0000) 2024-02-20T09:29:17.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:17.278+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.278960+0000) 2024-02-20T09:29:17.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:17.298+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.299163+0000) 2024-02-20T09:29:17.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:17.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:17.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:17.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:17.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:17.418+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:17.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:17.419+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:17.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:17.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:18.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:17.777+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:18.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:17.778+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:17.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:17.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:17.798+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:17.798+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:18.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:17.987+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.987953+0000) 2024-02-20T09:29:18.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:18.193+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66286 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:18.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:18.189+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.190712+0000) 2024-02-20T09:29:18.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:18.188+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.189378+0000) 2024-02-20T09:29:18.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:18.278+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.279559+0000) 2024-02-20T09:29:18.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:18.188+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.189626+0000) 2024-02-20T09:29:18.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:18.298+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.299711+0000) 2024-02-20T09:29:18.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:18.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:18.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:18.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:18.469 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:18.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:18.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:18.392+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:18.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:18.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:18.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:19.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:18.747+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:18.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:18.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:18.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:18.846+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:18.987+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.988855+0000) 2024-02-20T09:29:19.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:19.281+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:19.282006+0000) 2024-02-20T09:29:19.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:19.302+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:19.302967+0000) 2024-02-20T09:29:19.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:19.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:19.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:19.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:19.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:19.434+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:19.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:19.435+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:19.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:20.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:19.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:19.786+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:20.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:19.787+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:19.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:19.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:19.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:19.862+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:20.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:19.990+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:19.991789+0000) 2024-02-20T09:29:20.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:20.283+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:20.284425+0000) 2024-02-20T09:29:20.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:20.304+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:20.305492+0000) 2024-02-20T09:29:20.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:20.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:20.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:20.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:20.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:20.403+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:20.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:20.405+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:20.782+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:21.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:20.782+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:20.782+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:21.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:20.783+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:20.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:20.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:20.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:20.875+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:21.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:20.993+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:20.994333+0000) 2024-02-20T09:29:21.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:21.287+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:21.288063+0000) 2024-02-20T09:29:21.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:21.308+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:21.308968+0000) 2024-02-20T09:29:21.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:21.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:21.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:21.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:21.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:21.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:21.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:21.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:21.847 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:21.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:21.848 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:21.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:21.848 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:21.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:21.848 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:21.736+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:21.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:22.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:21.845+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:21.846+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:21.846+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:22.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:21.995+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:21.996833+0000) 2024-02-20T09:29:22.496 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:22.290+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.291640+0000) 2024-02-20T09:29:22.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:22.310+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.311446+0000) 2024-02-20T09:29:22.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:22.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:22.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:22.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:22.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:22.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:22.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:22.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:22.831 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:22.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:22.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:22.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:22.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:22.721+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:22.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:22.722+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:23.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:22.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:23.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:22.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:23.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:22.830+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:23.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:22.830+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:23.197 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:22.998+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.999410+0000) 2024-02-20T09:29:23.468 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:23.201+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66326 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:23.469 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:23.199+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.199991+0000) 2024-02-20T09:29:23.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:23.198+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.199759+0000) 2024-02-20T09:29:23.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:23.291+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.292578+0000) 2024-02-20T09:29:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:23.196+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.197725+0000) 2024-02-20T09:29:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:23.311+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.312336+0000) 2024-02-20T09:29:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:23.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:23.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:23.407+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:23.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:23.408+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:24.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:23.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:24.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:23.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:24.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:23.735+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:24.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:23.736+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:24.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:23.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:23.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:23.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:23.875+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:24.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:23.999+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:24.000022+0000) 2024-02-20T09:29:24.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:24.314+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:24.314930+0000) 2024-02-20T09:29:24.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:24.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:24.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:24.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:24.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:24.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:24.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:24.394+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:24.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:24.293+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:24.294702+0000) 2024-02-20T09:29:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:24.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:24.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:24.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:24.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:24.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:24.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:24.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:24.926+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:25.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:25.001+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:25.002497+0000) 2024-02-20T09:29:25.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:25.296+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:25.297271+0000) 2024-02-20T09:29:25.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:25.316+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:25.317379+0000) 2024-02-20T09:29:25.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:25.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:25.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:25.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:25.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:25.432+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:25.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:25.433+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:25.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:25.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:25.700+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:25.700+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:25.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:25.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:25.886+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:25.887+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:26.004+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:26.005121+0000) 2024-02-20T09:29:26.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:26.298+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:26.299605+0000) 2024-02-20T09:29:26.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:26.318+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:26.319635+0000) 2024-02-20T09:29:26.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:26.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:26.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:26.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:26.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:26.475+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:26.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:26.476+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:26.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:26.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:26.724+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:26.725+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:26.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:26.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:26.892+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:26.893+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:27.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:27.007+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.007981+0000) 2024-02-20T09:29:27.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:27.301+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.302213+0000) 2024-02-20T09:29:27.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:27.321+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.322033+0000) 2024-02-20T09:29:27.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:27.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:27.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:27.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:27.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:27.463+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:27.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:27.464+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:27.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:27.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:27.733+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:27.734+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:27.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:27.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:27.866+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:27.867+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:28.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:28.209+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66365 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:28.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:28.205+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.206533+0000) 2024-02-20T09:29:28.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:28.206+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.207152+0000) 2024-02-20T09:29:28.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:28.009+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.010736+0000) 2024-02-20T09:29:28.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:28.206+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.207054+0000) 2024-02-20T09:29:28.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:28.302+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.302959+0000) 2024-02-20T09:29:28.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:28.322+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.322946+0000) 2024-02-20T09:29:28.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:28.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:28.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:28.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:28.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:28.433+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:28.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:28.434+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:28.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:28.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:28.751+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:28.752+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:28.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:28.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:28.900+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:28.901+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:29.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:29.010+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:29.011669+0000) 2024-02-20T09:29:29.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:29.304+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:29.305505+0000) 2024-02-20T09:29:29.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:29.324+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:29.325523+0000) 2024-02-20T09:29:29.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:29.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:29.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:29.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:29.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:29.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:29.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:29.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:29.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:29.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:29.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:29.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:29.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:29.746+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:29.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:29.747+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:29.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:29.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:29.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:29.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:29.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:29.940+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:29.941+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:30.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:30.014+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:30.015212+0000) 2024-02-20T09:29:30.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:30.326+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:30.327639+0000) 2024-02-20T09:29:30.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:30.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:30.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:30.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:30.559 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:30.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:30.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:30.426+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:30.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:30.308+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:30.309071+0000) 2024-02-20T09:29:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:30.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:30.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:30.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:30.723+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:30.724+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:31.252 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:30.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:31.253 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:30.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:31.253 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:30.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:31.253 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:30.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:31.253 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:31.016+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:31.017660+0000) 2024-02-20T09:29:31.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:31.310+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:31.311402+0000) 2024-02-20T09:29:31.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:31.329+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:31.330190+0000) 2024-02-20T09:29:31.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:31.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:31.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:31.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:31.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:31.466+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:31.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:31.467+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:31.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:31.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:31.759+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:31.760+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:31.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:31.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:31.933+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:31.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:31.934+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:32.255 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:32.019+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.020246+0000) 2024-02-20T09:29:32.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:32.312+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.313822+0000) 2024-02-20T09:29:32.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:32.331+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.332861+0000) 2024-02-20T09:29:32.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:32.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:32.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:32.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:32.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:32.457+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:32.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:32.458+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:32.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:32.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:32.785+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:32.786+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:33.238 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:33.217+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66405 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:33.238 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:33.213+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.214629+0000) 2024-02-20T09:29:33.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:33.213+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.214089+0000) 2024-02-20T09:29:33.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:32.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:33.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:32.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:33.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:32.975+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:33.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:32.976+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:33.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:33.022+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.022958+0000) 2024-02-20T09:29:33.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:33.215+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.215931+0000) 2024-02-20T09:29:33.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:33.313+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.314698+0000) 2024-02-20T09:29:33.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:33.332+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.333417+0000) 2024-02-20T09:29:33.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:33.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:33.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:33.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:33.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:33.482+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:33.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:33.483+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:33.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:33.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:33.800+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:33.801+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:34.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:34.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:34.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:34.019+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:34.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:34.020+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:34.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:34.022+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:34.023891+0000) 2024-02-20T09:29:34.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:34.316+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:34.317192+0000) 2024-02-20T09:29:34.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:34.335+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:34.336891+0000) 2024-02-20T09:29:34.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:34.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:34.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:34.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:34.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:34.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:34.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:34.436+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:34.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:34.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:34.839+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:34.840+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:35.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:35.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:35.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:35.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:35.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:35.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:35.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:35.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:35.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:35.026+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:35.027491+0000) 2024-02-20T09:29:35.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:35.319+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:35.320548+0000) 2024-02-20T09:29:35.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:35.338+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:35.339106+0000) 2024-02-20T09:29:35.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:35.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:35.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:35.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:35.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:35.474+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:35.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:35.475+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:35.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:35.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:35.886+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:35.887+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:36.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:36.030+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:36.030956+0000) 2024-02-20T09:29:36.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:36.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:36.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:36.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:36.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:36.065+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:36.324 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:36.066+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:36.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:36.322+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:36.323184+0000) 2024-02-20T09:29:36.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:36.340+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:36.341591+0000) 2024-02-20T09:29:36.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:36.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:36.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:36.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:36.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:36.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:36.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:36.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:36.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:36.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:36.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:36.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:36.881+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:37.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:37.032+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.033396+0000) 2024-02-20T09:29:37.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:37.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:37.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:37.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:37.327 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:37.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:37.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:37.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:37.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:37.325+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.326017+0000) 2024-02-20T09:29:37.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:37.343+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.344061+0000) 2024-02-20T09:29:37.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:37.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:37.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:37.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:37.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:37.436+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:37.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:37.437+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:37.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:37.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:37.854+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:37.855+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:38.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:38.225+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66444 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:38.328 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:38.221+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.222368+0000) 2024-02-20T09:29:38.328 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:38.035+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.035979+0000) 2024-02-20T09:29:38.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:38.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:38.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:38.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:38.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:38.046+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:38.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:38.047+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:38.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:38.222+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.223309+0000) 2024-02-20T09:29:38.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:38.221+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.222544+0000) 2024-02-20T09:29:38.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:38.325+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.326873+0000) 2024-02-20T09:29:38.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:38.343+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.344797+0000) 2024-02-20T09:29:38.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:38.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:38.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:38.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:38.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:38.454+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:38.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:38.455+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:38.853+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:38.853+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:38.853+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:38.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:38.854+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:39.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:39.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:39.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:39.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:39.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:39.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:39.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:39.013+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:39.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:39.035+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:39.036643+0000) 2024-02-20T09:29:39.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:39.328+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:39.329409+0000) 2024-02-20T09:29:39.688 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:39.345+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:39.346925+0000) 2024-02-20T09:29:39.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:39.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:39.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:39.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:39.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:39.479+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:39.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:39.480+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:39.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:39.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:39.842+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:39.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:39.844+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:40.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:40.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:40.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:40.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:40.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:40.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:40.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:40.032+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:40.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:40.039+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:40.040153+0000) 2024-02-20T09:29:40.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:40.331+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:40.331952+0000) 2024-02-20T09:29:40.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:40.348+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:40.348953+0000) 2024-02-20T09:29:40.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:40.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:40.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:40.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:40.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:40.458+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:40.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:40.459+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:40.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:40.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:40.867+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:40.868+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:41.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:41.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:41.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:41.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:41.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:41.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:41.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:41.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:41.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:41.041+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:41.042844+0000) 2024-02-20T09:29:41.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:41.334+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:41.335250+0000) 2024-02-20T09:29:41.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:41.350+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:41.351519+0000) 2024-02-20T09:29:41.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:41.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:41.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:41.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:41.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:41.419+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:41.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:41.420+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:41.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:41.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:41.845+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:41.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:41.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:42.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:42.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:42.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:42.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:42.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:42.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:42.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:42.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:42.265 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:42.045+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.046346+0000) 2024-02-20T09:29:42.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:42.336+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.337723+0000) 2024-02-20T09:29:42.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:42.353+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.354024+0000) 2024-02-20T09:29:42.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:42.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:42.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:42.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:42.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:42.435+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:42.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:42.437+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:42.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:42.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:42.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:42.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:42.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:42.881+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:42.882+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:43.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:43.230+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.230962+0000) 2024-02-20T09:29:43.340 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:43.229+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.230571+0000) 2024-02-20T09:29:43.340 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:43.234+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66484 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:43.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:43.048+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.049924+0000) 2024-02-20T09:29:43.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:43.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:43.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:43.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:43.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:43.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:43.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:43.071+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:43.341 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:43.230+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.231596+0000) 2024-02-20T09:29:43.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:43.337+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.338724+0000) 2024-02-20T09:29:43.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:43.354+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.354973+0000) 2024-02-20T09:29:43.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:43.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:43.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:43.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:43.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:43.448+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:43.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:43.449+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:43.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:43.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:43.892+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:43.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:43.893+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:44.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:44.049+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:44.050533+0000) 2024-02-20T09:29:44.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:44.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:44.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:44.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:44.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:44.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:44.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:44.100+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:44.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:44.340+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:44.341430+0000) 2024-02-20T09:29:44.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:44.356+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:44.357567+0000) 2024-02-20T09:29:44.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:44.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:44.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:44.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:44.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:44.445+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:44.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:44.446+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:45.051+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:45.052641+0000) 2024-02-20T09:29:45.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:45.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:45.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:45.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:45.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:44.939+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:44.939+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:44.939+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:44.940+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:45.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:45.344+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:45.344934+0000) 2024-02-20T09:29:45.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:45.358+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:45.359721+0000) 2024-02-20T09:29:45.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:45.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:45.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:45.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:45.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:45.481+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:45.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:45.482+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:45.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:45.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:45.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:45.912+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:45.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:45.913+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:46.054+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:46.055046+0000) 2024-02-20T09:29:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:46.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:46.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:46.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:46.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:46.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:46.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:46.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:46.347+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:46.348524+0000) 2024-02-20T09:29:46.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:46.361+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:46.362391+0000) 2024-02-20T09:29:46.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:46.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:46.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:46.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:46.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:46.511+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:46.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:46.512+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:46.889+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:46.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:46.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:46.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:46.890+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:47.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:47.056+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.057892+0000) 2024-02-20T09:29:47.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:47.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:47.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:47.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:47.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:47.124+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:47.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:47.125+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:47.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:47.351+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.352026+0000) 2024-02-20T09:29:47.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:47.363+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.364786+0000) 2024-02-20T09:29:47.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:47.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:47.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:47.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:47.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:47.478+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:47.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:47.479+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:47.863+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:47.863+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:47.863+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:47.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:47.864+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:48.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:48.242+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66523 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:48.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:48.238+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.239911+0000) 2024-02-20T09:29:48.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:48.238+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.239789+0000) 2024-02-20T09:29:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:48.060+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.061513+0000) 2024-02-20T09:29:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:48.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:48.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:48.096+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:48.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:48.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:48.240+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.241637+0000) 2024-02-20T09:29:48.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:48.352+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.353025+0000) 2024-02-20T09:29:48.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:48.364+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.365678+0000) 2024-02-20T09:29:48.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:48.527+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:48.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:48.527+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:48.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:48.527+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:48.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:48.528+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:48.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:48.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:48.818+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:48.819+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:49.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:49.061+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:49.062297+0000) 2024-02-20T09:29:49.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:49.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:49.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:49.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:49.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:49.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:49.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:49.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:49.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:49.354+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:49.355227+0000) 2024-02-20T09:29:49.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:49.366+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:49.367730+0000) 2024-02-20T09:29:49.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:49.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:49.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:49.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:49.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:49.577+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:49.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:49.578+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:49.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:49.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:49.829+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:49.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:49.830+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:50.359 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:50.063+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:50.065617+0000) 2024-02-20T09:29:50.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:50.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:50.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:50.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:50.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:50.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:50.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:50.073+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:50.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:50.356+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:50.358792+0000) 2024-02-20T09:29:50.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:50.367+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:50.369879+0000) 2024-02-20T09:29:50.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:50.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:50.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:50.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:50.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:50.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:50.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:50.591+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:50.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:50.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:50.794+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:50.795+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:51.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:51.065+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:51.067746+0000) 2024-02-20T09:29:51.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:51.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:51.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:51.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:51.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:51.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:51.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:51.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:51.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:51.360+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:51.362273+0000) 2024-02-20T09:29:51.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:51.370+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:51.372563+0000) 2024-02-20T09:29:51.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:51.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:51.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:51.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:51.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:51.596+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:51.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:51.597+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:51.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:51.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:51.843+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:51.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:51.844+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:52.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:52.068+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.070374+0000) 2024-02-20T09:29:52.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:52.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:52.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:52.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:52.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:52.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:52.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:52.157+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:52.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:52.374+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.376015+0000) 2024-02-20T09:29:52.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:52.363+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.365652+0000) 2024-02-20T09:29:52.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:52.557+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:52.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:52.557+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:52.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:52.557+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:52.849 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:52.558+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:53.147 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:52.846+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:53.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:52.846+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:53.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:52.846+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:53.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:52.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:53.148 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:53.071+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.072644+0000) 2024-02-20T09:29:53.439 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:53.251+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66563 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:53.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:53.247+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.248925+0000) 2024-02-20T09:29:53.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:53.375+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.376634+0000) 2024-02-20T09:29:53.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:53.247+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.248492+0000) 2024-02-20T09:29:53.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:53.365+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.366435+0000) 2024-02-20T09:29:53.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:53.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:53.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:53.145+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:53.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:53.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:53.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:53.147+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:53.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:53.247+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.248303+0000) 2024-02-20T09:29:53.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:53.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:53.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:53.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:53.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:53.524+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:53.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:53.525+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:53.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:53.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:53.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:53.834+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:53.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:53.835+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:54.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:54.072+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:54.073447+0000) 2024-02-20T09:29:54.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:54.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:54.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:54.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:54.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:54.180+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:54.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:54.181+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:54.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:54.367+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:54.368320+0000) 2024-02-20T09:29:54.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:54.378+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:54.378973+0000) 2024-02-20T09:29:54.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:54.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:54.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:54.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:54.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:54.490+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:54.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:54.491+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:54.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:54.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:54.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:54.880+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:54.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:54.880+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:55.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:55.075+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:55.076741+0000) 2024-02-20T09:29:55.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:55.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:55.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:55.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:55.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:55.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:55.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:55.171+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:55.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:55.369+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:55.370690+0000) 2024-02-20T09:29:55.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:55.380+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:55.381114+0000) 2024-02-20T09:29:55.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:55.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:55.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:55.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:55.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:55.494+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:55.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:55.495+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:55.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:55.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:55.869+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:55.870+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:56.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:56.077+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:56.078784+0000) 2024-02-20T09:29:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:56.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:56.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:56.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:56.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:56.187+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:56.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:56.372+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:56.373326+0000) 2024-02-20T09:29:56.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:56.382+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:56.383230+0000) 2024-02-20T09:29:56.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:56.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:56.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:56.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:56.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:56.544+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:56.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:56.545+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:56.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:56.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:56.901+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:56.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:56.902+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:57.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:57.080+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.081361+0000) 2024-02-20T09:29:57.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:57.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:57.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:57.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:57.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:57.226+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:57.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:57.227+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:57.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:57.375+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.376683+0000) 2024-02-20T09:29:57.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:57.385+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.386817+0000) 2024-02-20T09:29:57.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:57.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:57.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:57.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:57.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:57.587+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:57.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:57.587+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:57.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:57.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:57.890+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:57.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:57.890+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:58.367 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:29:58.259+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66603 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:29:58.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:58.256+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.256993+0000) 2024-02-20T09:29:58.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:58.255+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.256470+0000) 2024-02-20T09:29:58.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:58.083+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.084824+0000) 2024-02-20T09:29:58.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:58.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:58.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:58.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:58.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:58.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:58.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:58.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:58.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:58.255+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.256711+0000) 2024-02-20T09:29:58.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:58.376+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.377639+0000) 2024-02-20T09:29:58.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:58.386+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.387816+0000) 2024-02-20T09:29:58.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:58.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:58.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:58.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:58.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:58.603+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:58.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:58.603+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:58.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:58.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:58.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:58.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:59.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:59.084+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:59.085480+0000) 2024-02-20T09:29:59.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:59.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:59.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:59.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:59.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:59.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:29:59.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:29:59.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:59.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:59.380+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:59.381199+0000) 2024-02-20T09:29:59.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:59.390+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:59.391096+0000) 2024-02-20T09:29:59.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:59.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:59.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:59.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:59.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:59.607+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:29:59.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:29:59.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:29:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:59.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:59.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:59.888+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:29:59.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:29:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:29:59.889+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:00.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:00.087+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:00.088717+0000) 2024-02-20T09:30:00.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:00.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:00.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:00.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:00.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:00.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:00.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:00.202+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:00.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:00.383+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:00.384784+0000) 2024-02-20T09:30:00.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:00.393+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:00.394429+0000) 2024-02-20T09:30:00.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:00.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:00.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:00.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:00.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:00.606+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:00.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:00.608+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:00.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:00.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:00.844+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:00.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:00.846+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:01.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:01.091+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:01.092171+0000) 2024-02-20T09:30:01.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:01.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:01.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:01.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:01.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:01.176+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:01.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:01.177+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:01.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:01.386+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:01.387054+0000) 2024-02-20T09:30:01.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:01.396+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:01.397889+0000) 2024-02-20T09:30:01.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:01.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:01.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:01.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:01.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:01.589+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:01.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:01.589+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:01.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:01.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:01.857+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:01.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:01.858+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:02.283 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:02.093+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.094616+0000) 2024-02-20T09:30:02.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:02.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:02.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:02.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:02.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:02.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:02.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:02.174+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:02.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:02.388+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.389865+0000) 2024-02-20T09:30:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:02.399+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.400525+0000) 2024-02-20T09:30:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:02.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:02.819 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:02.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:02.819 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:02.539+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:02.819 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:02.541+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:03.098 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:02.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:03.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:02.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:03.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:02.817+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:03.099 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:02.818+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:03.368 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:03.267+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66642 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:03.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:03.264+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.265581+0000) 2024-02-20T09:30:03.369 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:03.263+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.264632+0000) 2024-02-20T09:30:03.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:03.097+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.098031+0000) 2024-02-20T09:30:03.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:03.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:03.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:03.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:03.178+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:03.179+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:03.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:03.264+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.265222+0000) 2024-02-20T09:30:03.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:03.389+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.390491+0000) 2024-02-20T09:30:03.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:03.400+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.401293+0000) 2024-02-20T09:30:03.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:03.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:03.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:03.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:03.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:03.509+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:03.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:03.510+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:03.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:03.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:03.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:03.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:03.814+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:03.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:03.815+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:04.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:04.098+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:04.099093+0000) 2024-02-20T09:30:04.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:04.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:04.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:04.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:04.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:04.157+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:04.352 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:04.158+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:04.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:04.391+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:04.392601+0000) 2024-02-20T09:30:04.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:04.402+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:04.403433+0000) 2024-02-20T09:30:04.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:04.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:04.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:04.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:04.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:04.471+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:04.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:04.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:04.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:04.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:04.813+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:04.814+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:05.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:05.100+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:05.101305+0000) 2024-02-20T09:30:05.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:05.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:05.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:05.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:05.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:05.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:05.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:05.171+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:05.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:05.395+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:05.395952+0000) 2024-02-20T09:30:05.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:05.405+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:05.406818+0000) 2024-02-20T09:30:05.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:05.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:05.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:05.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:05.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:05.438+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:05.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:05.439+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:05.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:05.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:05.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:05.787+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:05.788+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:06.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:06.102+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:06.103486+0000) 2024-02-20T09:30:06.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:06.172+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:06.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:06.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:06.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:06.173+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:06.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:06.173+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:06.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:06.397+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:06.398375+0000) 2024-02-20T09:30:06.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:06.409+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:06.410388+0000) 2024-02-20T09:30:06.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:06.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:06.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:06.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:06.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:06.455+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:06.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:06.456+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:06.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:06.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:06.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:06.753+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:06.754+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:07.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:07.105+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.106859+0000) 2024-02-20T09:30:07.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:07.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:07.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:07.209+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:07.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:07.210+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:07.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:07.411+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.412603+0000) 2024-02-20T09:30:07.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:07.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:07.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:07.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:07.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:07.417+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:07.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:07.418+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:07.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:07.399+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.400586+0000) 2024-02-20T09:30:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:07.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:07.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:07.781+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:07.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:07.782+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:08.369 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:08.276+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66682 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:08.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:08.273+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.274705+0000) 2024-02-20T09:30:08.369 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:08.274+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.275211+0000) 2024-02-20T09:30:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:08.108+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.109366+0000) 2024-02-20T09:30:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:08.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:08.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:08.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:08.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:08.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:08.272+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.273662+0000) 2024-02-20T09:30:08.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:08.400+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.401664+0000) 2024-02-20T09:30:08.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:08.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:08.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:08.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:08.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:08.383+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:08.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:08.384+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:08.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:08.412+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.413636+0000) 2024-02-20T09:30:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:08.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:08.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:08.745+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:08.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:08.746+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:09.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:09.109+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:09.110259+0000) 2024-02-20T09:30:09.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:09.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:09.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:09.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:09.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:09.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:09.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:09.286+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:09.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:09.414+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:09.415746+0000) 2024-02-20T09:30:09.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:09.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:09.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:09.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:09.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:09.421+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:09.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:09.422+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:09.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:09.403+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:09.404560+0000) 2024-02-20T09:30:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:09.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:09.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:09.734+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:09.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:09.735+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:10.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:10.112+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:10.113630+0000) 2024-02-20T09:30:10.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:10.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:10.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:10.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:10.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:10.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:10.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:10.307+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:10.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:10.407+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:10.408108+0000) 2024-02-20T09:30:10.691 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:10.418+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:10.419014+0000) 2024-02-20T09:30:10.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:10.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:10.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:10.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:10.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:10.420+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:10.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:10.421+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:10.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:10.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:10.703+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:10.704+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:11.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:11.114+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:11.115895+0000) 2024-02-20T09:30:11.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:11.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:11.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:11.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:11.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:11.346+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:11.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:11.347+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:11.689 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:11.409+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:11.410264+0000) 2024-02-20T09:30:11.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:11.421+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:11.422501+0000) 2024-02-20T09:30:11.689 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:11.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:11.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:11.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:11.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:11.423+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:11.690 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:11.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:11.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:11.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:11.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:11.687+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:11.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:11.688+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:12.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:12.118+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.119298+0000) 2024-02-20T09:30:12.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:12.412+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.412944+0000) 2024-02-20T09:30:12.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:12.423+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.424678+0000) 2024-02-20T09:30:12.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:12.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:12.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:12.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:12.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:12.451+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:12.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:12.452+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:12.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:12.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:12.389+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:12.390+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:12.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:12.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:12.648+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:12.649+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:13.406 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:13.284+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66721 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:13.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:13.280+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.281384+0000) 2024-02-20T09:30:13.407 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:13.281+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.282573+0000) 2024-02-20T09:30:13.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:13.120+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.121770+0000) 2024-02-20T09:30:13.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:13.279+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.280826+0000) 2024-02-20T09:30:13.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:13.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:13.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:13.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:13.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:13.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:13.408 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:13.349+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:13.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:13.424+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.425696+0000) 2024-02-20T09:30:13.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:13.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:13.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:13.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:13.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:13.484+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:13.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:13.485+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:13.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:13.412+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.413753+0000) 2024-02-20T09:30:13.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:13.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:13.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:13.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:13.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:13.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:13.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:13.601+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:14.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:14.121+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:14.122845+0000) 2024-02-20T09:30:14.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:14.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:14.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:14.332+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:14.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:14.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:14.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:14.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:14.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:14.428+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:14.429134+0000) 2024-02-20T09:30:14.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:14.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:14.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:14.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:14.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:14.495+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:14.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:14.496+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:14.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:14.415+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:14.416023+0000) 2024-02-20T09:30:14.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:14.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:14.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:14.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:14.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:14.625+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:14.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:14.626+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:15.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:15.124+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:15.125116+0000) 2024-02-20T09:30:15.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:15.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:15.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:15.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:15.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:15.367+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:15.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:15.368+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:15.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:15.431+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:15.432532+0000) 2024-02-20T09:30:15.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:15.546+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:15.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:15.546+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:15.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:15.546+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:15.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:15.547+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:15.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:15.418+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:15.419303+0000) 2024-02-20T09:30:15.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:15.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:15.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:15.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:15.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:15.658+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:15.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:15.659+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:16.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:16.126+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:16.127254+0000) 2024-02-20T09:30:16.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:16.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:16.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:16.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:16.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:16.377+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:16.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:16.433+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:16.434881+0000) 2024-02-20T09:30:16.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:16.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:16.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:16.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:16.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:16.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:16.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:16.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:16.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:16.420+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:16.421831+0000) 2024-02-20T09:30:16.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:16.378+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:16.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:16.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:16.638+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:16.639+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:17.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:17.129+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.130563+0000) 2024-02-20T09:30:17.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:17.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:17.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:17.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:17.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:17.373+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:17.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:17.374+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:17.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:17.436+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.437770+0000) 2024-02-20T09:30:17.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:17.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:17.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:17.590+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:17.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:17.590+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:17.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:17.423+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.424585+0000) 2024-02-20T09:30:17.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:17.611+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:17.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:17.611+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:17.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:17.611+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:17.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:17.613+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:18.411 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:18.292+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66761 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:18.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:18.290+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.291319+0000) 2024-02-20T09:30:18.412 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:18.289+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.290107+0000) 2024-02-20T09:30:18.412 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:18.131+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.132882+0000) 2024-02-20T09:30:18.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:18.288+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.289923+0000) 2024-02-20T09:30:18.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:18.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:18.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:18.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:18.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:18.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:18.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:18.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:18.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:18.441+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.442130+0000) 2024-02-20T09:30:18.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:18.553+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:18.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:18.554+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:18.555+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:18.424+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.425607+0000) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:18.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:18.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:18.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:18.615+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:18.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:18.616+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:19.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:19.132+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:19.133607+0000) 2024-02-20T09:30:19.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:19.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:19.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:19.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:19.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:19.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:19.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:19.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:19.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:19.444+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:19.445811+0000) 2024-02-20T09:30:19.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:19.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:19.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:19.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:19.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:19.599+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:19.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:19.600+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:19.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:19.428+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:19.428952+0000) 2024-02-20T09:30:19.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:19.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:19.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:19.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:19.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:19.602+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:19.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:19.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:20.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:20.135+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:20.136119+0000) 2024-02-20T09:30:20.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:20.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:20.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:20.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:20.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:20.308+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:20.433 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:20.309+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:20.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:20.431+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:20.432098+0000) 2024-02-20T09:30:20.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:20.650+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:20.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:20.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:20.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:20.651+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:20.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:20.651+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:20.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:20.448+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:20.449131+0000) 2024-02-20T09:30:20.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:20.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:20.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:20.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:20.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:20.594+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:20.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:20.595+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:21.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:21.138+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:21.139461+0000) 2024-02-20T09:30:21.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:21.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:21.435 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:21.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:21.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:21.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:21.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:21.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:21.692 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:21.450+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:21.451276+0000) 2024-02-20T09:30:21.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:21.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:21.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:21.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:21.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:21.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:21.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:21.566+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:21.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:21.433+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:21.434411+0000) 2024-02-20T09:30:21.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:21.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:21.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:21.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:21.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:21.670+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:21.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:21.672+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:22.306 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:22.141+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.142865+0000) 2024-02-20T09:30:22.652 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:22.436+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.436970+0000) 2024-02-20T09:30:22.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:22.452+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.453400+0000) 2024-02-20T09:30:22.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:22.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:22.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:22.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:22.652 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:22.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:22.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:22.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:22.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:22.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:22.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:22.306+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:22.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:22.307+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:22.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:22.308+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:22.650+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:22.650+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:22.650+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:22.651+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:23.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:23.296+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.297713+0000) 2024-02-20T09:30:23.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:23.297+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.298822+0000) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:23.301+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66800 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:23.145+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.146269+0000) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:23.296+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.297392+0000) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:23.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:23.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:23.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:23.421 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:23.302+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:23.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:23.453+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.454322+0000) 2024-02-20T09:30:23.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:23.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:23.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:23.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:23.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:23.532+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:23.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:23.533+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:23.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:23.437+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.438016+0000) 2024-02-20T09:30:23.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:23.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:23.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:23.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:23.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:23.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:23.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:23.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:24.441 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:24.146+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:24.147358+0000) 2024-02-20T09:30:24.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:24.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:24.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:24.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:24.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:24.269+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:24.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:24.270+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:24.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:24.455+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:24.456203+0000) 2024-02-20T09:30:24.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:24.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:24.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:24.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:24.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:24.550+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:24.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:24.551+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:24.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:24.439+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:24.440455+0000) 2024-02-20T09:30:24.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:24.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:24.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:24.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:24.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:24.594+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:24.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:24.595+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:25.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:25.149+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:25.149976+0000) 2024-02-20T09:30:25.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:25.232+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:25.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:25.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:25.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:25.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:25.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:25.233+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:25.457+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:25.458348+0000) 2024-02-20T09:30:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:25.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:25.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:25.568+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:25.569+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:25.442+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:25.443665+0000) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:25.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:25.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:25.624+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:25.625+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:26.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:26.151+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:26.152733+0000) 2024-02-20T09:30:26.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:26.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:26.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:26.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:26.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:26.184+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:26.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:26.184+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:26.446+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:26.447014+0000) 2024-02-20T09:30:26.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:26.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:26.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:26.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:26.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:26.459+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:26.460573+0000) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:26.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:26.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:26.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:26.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:26.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:26.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:27.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:27.154+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.154954+0000) 2024-02-20T09:30:27.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:27.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:27.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:27.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:27.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:27.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:27.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:27.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:27.462+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.463024+0000) 2024-02-20T09:30:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:27.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:27.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:27.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:27.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:27.448+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.449705+0000) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:27.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:27.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:27.585+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:27.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:27.586+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:28.430 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:28.310+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66841 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:28.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:28.309+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.310422+0000) 2024-02-20T09:30:28.430 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:28.306+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.307810+0000) 2024-02-20T09:30:28.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:28.157+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.158343+0000) 2024-02-20T09:30:28.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:28.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:28.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:28.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:28.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:28.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:28.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:28.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:28.431 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:28.306+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.307286+0000) 2024-02-20T09:30:28.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:28.463+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.464140+0000) 2024-02-20T09:30:28.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:28.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:28.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:28.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:28.604+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:28.604+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:28.449+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.450738+0000) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:28.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:28.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:28.589+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:28.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:28.590+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:29.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:29.158+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:29.159445+0000) 2024-02-20T09:30:29.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:29.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:29.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:29.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:29.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:29.451+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:29.452883+0000) 2024-02-20T09:30:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:29.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:29.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:29.573+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:29.574+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:29.466+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:29.467425+0000) 2024-02-20T09:30:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:29.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:29.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:29.615+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:29.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:29.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:30.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:30.160+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:30.161501+0000) 2024-02-20T09:30:30.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:30.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:30.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:30.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:30.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:30.276+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:30.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:30.277+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:30.468+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:30.469601+0000) 2024-02-20T09:30:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:30.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:30.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:30.624+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:30.625+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:30.455+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:30.456460+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:30.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:30.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:30.614+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:30.615+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:31.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:31.163+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:31.164848+0000) 2024-02-20T09:30:31.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:31.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:31.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:31.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:31.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:31.247+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:31.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:31.248+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:31.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:31.457+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:31.458472+0000) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:31.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:31.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:31.570+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:31.571+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:31.470+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:31.471638+0000) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:31.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:31.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:31.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:31.588+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:31.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:31.588+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:32.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:32.165+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.166893+0000) 2024-02-20T09:30:32.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:32.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:32.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:32.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:32.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:32.217+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:32.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:32.218+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:32.588 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:32.460+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.460954+0000) 2024-02-20T09:30:32.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:32.472+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.473793+0000) 2024-02-20T09:30:32.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:32.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:32.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:32.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:32.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:32.556+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:32.589 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:32.557+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:32.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:32.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:32.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:32.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:32.587+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:33.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:33.314+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.315885+0000) 2024-02-20T09:30:33.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:33.315+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.316298+0000) 2024-02-20T09:30:33.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:33.318+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66880 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:33.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:33.168+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.169653+0000) 2024-02-20T09:30:33.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:33.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:33.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:33.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:33.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:33.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:33.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:33.214+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:33.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:33.314+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.315387+0000) 2024-02-20T09:30:33.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:33.473+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.474897+0000) 2024-02-20T09:30:33.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:33.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:33.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:33.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:33.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:33.529+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:33.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:33.530+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:33.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:33.461+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.462051+0000) 2024-02-20T09:30:33.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:33.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:33.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:33.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:33.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:33.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:33.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:33.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:34.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:34.169+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:34.170397+0000) 2024-02-20T09:30:34.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:34.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:34.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:34.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:34.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:34.179+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:34.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:34.180+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:34.464+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:34.465622+0000) 2024-02-20T09:30:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:34.604+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:34.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:34.604+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:34.604+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:34.605+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:34.476+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:34.477127+0000) 2024-02-20T09:30:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:34.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:34.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:34.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:34.579+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:34.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:34.580+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:35.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:35.173+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:35.173969+0000) 2024-02-20T09:30:35.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:35.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:35.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:35.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:35.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:35.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:35.326 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:35.203+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:35.618 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:35.478+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:35.479875+0000) 2024-02-20T09:30:35.618 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:35.467+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:35.468446+0000) 2024-02-20T09:30:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:35.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:35.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:35.621+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:35.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:35.622+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:35.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:35.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:35.616+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:35.617+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:36.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:36.176+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:36.177285+0000) 2024-02-20T09:30:36.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:36.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:36.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:36.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:36.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:36.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:36.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:36.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:36.481+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:36.481971+0000) 2024-02-20T09:30:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:36.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:36.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:36.584+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:36.585+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:36.470+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:36.470956+0000) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:36.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:36.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:36.627+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:36.628+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:37.444 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:37.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:37.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:37.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:37.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:37.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:37.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:37.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:37.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:37.179+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.180923+0000) 2024-02-20T09:30:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:37.483+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.484183+0000) 2024-02-20T09:30:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:37.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:37.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:37.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:37.570+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:37.571+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:37.473+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.474356+0000) 2024-02-20T09:30:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:37.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:37.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:37.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:37.584+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:37.585+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:38.444 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:38.327+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66920 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:38.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:38.322+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.323486+0000) 2024-02-20T09:30:38.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:38.322+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.323798+0000) 2024-02-20T09:30:38.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:38.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:38.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:38.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:38.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:38.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:38.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:38.151+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:38.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:38.182+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.183191+0000) 2024-02-20T09:30:38.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:38.324+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.325211+0000) 2024-02-20T09:30:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:38.474+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.475511+0000) 2024-02-20T09:30:38.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:38.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:38.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:38.548+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:38.549+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:38.484+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.485314+0000) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:38.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:38.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:38.574+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:38.575+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:39.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:39.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:39.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:39.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:39.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:39.129+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:39.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:39.130+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:39.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:39.183+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:39.184228+0000) 2024-02-20T09:30:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:39.487+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:39.488726+0000) 2024-02-20T09:30:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:39.566+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:39.566+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:39.566+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:39.568+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:39.477+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:39.478067+0000) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:39.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:39.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:39.532+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:39.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:39.533+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:40.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:40.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:40.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:40.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:40.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:40.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:40.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:40.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:40.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:40.185+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:40.186253+0000) 2024-02-20T09:30:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:40.491+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:40.492152+0000) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:40.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:40.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:40.545+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:40.547+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:40.479+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:40.480482+0000) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:40.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:40.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:40.489+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:40.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:40.490+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:41.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:41.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:41.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:41.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:41.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:41.142+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:41.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:41.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:41.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:41.188+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:41.189510+0000) 2024-02-20T09:30:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:41.482+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:41.483022+0000) 2024-02-20T09:30:41.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:41.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:41.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:41.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:41.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:41.494+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:41.494952+0000) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:41.503+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:41.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:41.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:41.504+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:41.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:41.505+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:42.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:42.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:42.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:42.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:42.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:42.156+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:42.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:42.157+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:42.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:42.191+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.192093+0000) 2024-02-20T09:30:42.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:42.484+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.486650+0000) 2024-02-20T09:30:42.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:42.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:42.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:42.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:42.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:42.549+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:42.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:42.550+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:42.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:42.496+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.498544+0000) 2024-02-20T09:30:42.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:42.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:42.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:42.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:42.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:42.519+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:42.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:42.520+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:43.446 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:43.334+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66959 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:43.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:43.330+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.332524+0000) 2024-02-20T09:30:43.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:43.330+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.332745+0000) 2024-02-20T09:30:43.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:43.137+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:43.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:43.137+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:43.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:43.137+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:43.447 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:43.138+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:43.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:43.192+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.194286+0000) 2024-02-20T09:30:43.448 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:43.331+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.333484+0000) 2024-02-20T09:30:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:43.485+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.487823+0000) 2024-02-20T09:30:43.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:43.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:43.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:43.586+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:43.587+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:43.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:43.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:43.472+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:43.472+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:43.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:43.497+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.499727+0000) 2024-02-20T09:30:44.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:44.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:44.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:44.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:44.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:44.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:44.196 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:44.089+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:44.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:44.193+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:44.194970+0000) 2024-02-20T09:30:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:44.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:44.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:44.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:44.487+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:44.488+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:44.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:44.499+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:44.501821+0000) 2024-02-20T09:30:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:44.488+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:44.490262+0000) 2024-02-20T09:30:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:44.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:44.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:44.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:44.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:44.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:45.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:45.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:45.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:45.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:45.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:45.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:45.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:45.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:45.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:45.195+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:45.197444+0000) 2024-02-20T09:30:45.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:45.502+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:45.504799+0000) 2024-02-20T09:30:45.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:45.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:45.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:45.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:45.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:45.526+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:45.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:45.527+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:45.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:45.491+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:45.492672+0000) 2024-02-20T09:30:45.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:45.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:45.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:45.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:45.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:45.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:45.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:45.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:46.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:46.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:46.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:46.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:46.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:46.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:46.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:46.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:46.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:46.200+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:46.201066+0000) 2024-02-20T09:30:46.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:46.494+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:46.495297+0000) 2024-02-20T09:30:46.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:46.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:46.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:46.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:46.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:46.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:46.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:46.508+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:46.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:46.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:46.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:46.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:46.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:46.486+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:46.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:46.487+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:46.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:46.506+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:46.507588+0000) 2024-02-20T09:30:47.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:47.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:47.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:47.010+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:47.011+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:47.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:47.204+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.204976+0000) 2024-02-20T09:30:47.695 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:47.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:47.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:47.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:47.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:47.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:47.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:47.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:47.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:47.509+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.510180+0000) 2024-02-20T09:30:47.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:47.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:47.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:47.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:47.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:47.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:47.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:47.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:47.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:47.496+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.497695+0000) 2024-02-20T09:30:48.338 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:48.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:48.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:48.002+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:48.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:48.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:48.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:48.003+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:48.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:48.206+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.207276+0000) 2024-02-20T09:30:48.695 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:48.343+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 66999 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:48.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:48.340+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.341039+0000) 2024-02-20T09:30:48.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:48.339+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.340576+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:48.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:48.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:48.410+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:48.411+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:48.510+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.511666+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:48.341+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.341980+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:48.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:48.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:48.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:48.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:48.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:48.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:48.433+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:48.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:48.497+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.498560+0000) 2024-02-20T09:30:49.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:49.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:49.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:49.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:49.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:49.013+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:49.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:49.014+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:49.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:49.207+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:49.208156+0000) 2024-02-20T09:30:49.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:49.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:49.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:49.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:49.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:49.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:49.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:49.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:49.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:49.500+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:49.500975+0000) 2024-02-20T09:30:49.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:49.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:49.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:49.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:49.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:49.425+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:49.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:49.427+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:49.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:49.514+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:49.514971+0000) 2024-02-20T09:30:50.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:50.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:50.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:50.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:50.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:50.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:50.209+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:50.210224+0000) 2024-02-20T09:30:50.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:50.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:50.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:50.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:50.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:50.389+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:50.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:50.389+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:50.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:50.516+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:50.517249+0000) 2024-02-20T09:30:50.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:50.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:50.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:50.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:50.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:50.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:50.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:50.416+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:50.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:50.503+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:50.504326+0000) 2024-02-20T09:30:51.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:51.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:51.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:51.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:51.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:51.034+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:51.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:51.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:51.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:51.212+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:51.213527+0000) 2024-02-20T09:30:51.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:51.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:51.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:51.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:51.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:51.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:51.519+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:51.520627+0000) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:51.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:51.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:51.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:51.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:51.374+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:51.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:51.505+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:51.506857+0000) 2024-02-20T09:30:52.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:52.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:52.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:52.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:52.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:52.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:52.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:52.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:52.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:52.215+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.216858+0000) 2024-02-20T09:30:52.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:52.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:52.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:52.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:52.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:52.402+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:52.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:52.403+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:52.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:52.522+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.523220+0000) 2024-02-20T09:30:52.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:52.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:52.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:52.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:52.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:52.398+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:52.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:52.400+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:52.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:52.509+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.510253+0000) 2024-02-20T09:30:53.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:53.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:53.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:53.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:53.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:53.084+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:53.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:53.085+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:53.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:53.219+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.220468+0000) 2024-02-20T09:30:53.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:53.347+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.348768+0000) 2024-02-20T09:30:53.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:53.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:53.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:53.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:53.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:53.452+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:53.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:53.452+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:53.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:53.523+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.524289+0000) 2024-02-20T09:30:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:53.348+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.349226+0000) 2024-02-20T09:30:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:53.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:53.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:53.435+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:53.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:53.436+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:53.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:53.510+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.511179+0000) 2024-02-20T09:30:53.698 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:53.351+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67038 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:53.698 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:53.350+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.351073+0000) 2024-02-20T09:30:54.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:54.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:54.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:54.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:54.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:54.115+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:54.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:54.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:54.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:54.220+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:54.221281+0000) 2024-02-20T09:30:54.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:54.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:54.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:54.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:54.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:54.470+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:54.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:54.471+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:54.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:54.526+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:54.527406+0000) 2024-02-20T09:30:54.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:54.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:54.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:54.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:54.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:54.454+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:54.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:54.455+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:54.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:54.512+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:54.513374+0000) 2024-02-20T09:30:55.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:55.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:55.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:55.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:55.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:55.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:55.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:55.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:55.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:55.222+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:55.223894+0000) 2024-02-20T09:30:55.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:55.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:55.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:55.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:55.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:55.431+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:55.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:55.432+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:55.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:55.529+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:55.530854+0000) 2024-02-20T09:30:55.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:55.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:55.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:55.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:55.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:55.421+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:55.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:55.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:55.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:55.516+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:55.516931+0000) 2024-02-20T09:30:56.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:56.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:56.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:56.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:56.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:56.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:56.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:56.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:56.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:56.225+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:56.225964+0000) 2024-02-20T09:30:56.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:56.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:56.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:56.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:56.422+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:56.423+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:56.533+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:56.534361+0000) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:56.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:56.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:56.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:56.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:56.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:56.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:56.518+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:56.518952+0000) 2024-02-20T09:30:57.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:57.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:57.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:57.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:57.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:57.071+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:57.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:57.072+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:57.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:57.227+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.228187+0000) 2024-02-20T09:30:57.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:57.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:57.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:57.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:57.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:57.439+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:57.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:57.441+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:57.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:57.536+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.537658+0000) 2024-02-20T09:30:57.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:57.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:57.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:57.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:57.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:57.441+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:57.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:57.442+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:57.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:57.521+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.522441+0000) 2024-02-20T09:30:58.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:58.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:58.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:58.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:58.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:58.047+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:58.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:58.048+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:58.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:58.229+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.230300+0000) 2024-02-20T09:30:58.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:58.356+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.357907+0000) 2024-02-20T09:30:58.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:58.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:58.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:58.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:58.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:58.424+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:58.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:58.425+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:58.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:58.537+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.538676+0000) 2024-02-20T09:30:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:58.357+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.358533+0000) 2024-02-20T09:30:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:58.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:58.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:58.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:58.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:58.522+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.523711+0000) 2024-02-20T09:30:58.699 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:30:58.360+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67078 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:30:58.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:58.356+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.357429+0000) 2024-02-20T09:30:59.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:59.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:59.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:59.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:59.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:59.033+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:30:59.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:59.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:59.352 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:30:59.230+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:59.231042+0000) 2024-02-20T09:30:59.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:59.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:59.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:59.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:59.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:59.391+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:30:59.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:59.391+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:30:59.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:30:59.541+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:59.542201+0000) 2024-02-20T09:30:59.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:59.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:59.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:59.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:59.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:59.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:30:59.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:59.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:59.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:30:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:30:59.525+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:59.526194+0000) 2024-02-20T09:31:00.352 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:00.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:00.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:00.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:00.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:00.035+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:00.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:00.036+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:00.353 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:00.233+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:00.234250+0000) 2024-02-20T09:31:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:00.528+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:00.529548+0000) 2024-02-20T09:31:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:00.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:00.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:00.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:00.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:00.534+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:00.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:00.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:00.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:00.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:00.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:00.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:00.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:00.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:00.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:00.543+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:00.544599+0000) 2024-02-20T09:31:01.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:01.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:01.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:01.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:01.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:01.303+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:01.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:01.304+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:01.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:01.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:01.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:01.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:01.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:01.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:01.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:01.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:01.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:01.236+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:01.236949+0000) 2024-02-20T09:31:01.696 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:01.545+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:01.546767+0000) 2024-02-20T09:31:01.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:01.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:01.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:01.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:01.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:01.527+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:01.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:01.528+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:01.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:01.532+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:01.532954+0000) 2024-02-20T09:31:02.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:02.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:02.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:02.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:02.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:02.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:02.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:02.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:02.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:02.238+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.239456+0000) 2024-02-20T09:31:02.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:02.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:02.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:02.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:02.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:02.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:02.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:02.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:02.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:02.548+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.549403+0000) 2024-02-20T09:31:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:02.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:02.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:02.499+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:02.500+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:02.535+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.536349+0000) 2024-02-20T09:31:03.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:03.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:03.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:03.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:03.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:03.318+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:03.357 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:03.319+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:03.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:03.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:03.091+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:03.092+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:03.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:03.241+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.242513+0000) 2024-02-20T09:31:03.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:03.364+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.365296+0000) 2024-02-20T09:31:03.697 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:03.368+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67116 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:03.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:03.366+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.367231+0000) 2024-02-20T09:31:03.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:03.549+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.550678+0000) 2024-02-20T09:31:03.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:03.365+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.365981+0000) 2024-02-20T09:31:03.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:03.536+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.537560+0000) 2024-02-20T09:31:03.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:03.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:03.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:03.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:03.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:03.538+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:03.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:03.539+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:04.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:04.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:04.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:04.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:04.375 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:04.356+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:04.376 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:04.357+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:04.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:04.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:04.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:04.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:04.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:04.121+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:04.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:04.121+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:04.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:04.242+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:04.243707+0000) 2024-02-20T09:31:04.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:04.552+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:04.553177+0000) 2024-02-20T09:31:04.697 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:04.539+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:04.540019+0000) 2024-02-20T09:31:04.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:04.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:04.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:04.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:04.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:04.543+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:04.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:04.544+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:05.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:05.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:05.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:05.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:05.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:05.323+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:05.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:05.324+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:05.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:05.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:05.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:05.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:05.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:05.160+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:05.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:05.161+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:05.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:05.245+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:05.246222+0000) 2024-02-20T09:31:05.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:05.555+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:05.556022+0000) 2024-02-20T09:31:05.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:05.541+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:05.542577+0000) 2024-02-20T09:31:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:05.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:05.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:05.567+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:05.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:05.568+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:06.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:06.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:06.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:06.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:06.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:06.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:06.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:06.314+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:06.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:06.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:06.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:06.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:06.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:06.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:06.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:06.194+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:06.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:06.248+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:06.249825+0000) 2024-02-20T09:31:06.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:06.557+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:06.558403+0000) 2024-02-20T09:31:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:06.544+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:06.545092+0000) 2024-02-20T09:31:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:06.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:06.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:06.603+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:06.603+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:07.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:07.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:07.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:07.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:07.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:07.285+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:07.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:07.286+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:07.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:07.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:07.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:07.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:07.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:07.194+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:07.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:07.195+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:07.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:07.251+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.252243+0000) 2024-02-20T09:31:07.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:07.561+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.561925+0000) 2024-02-20T09:31:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:07.547+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.548813+0000) 2024-02-20T09:31:07.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:07.596+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:07.596+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:07.596+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:07.598+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:08.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:08.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:08.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:08.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:08.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:08.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:08.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:08.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:08.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:08.373+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.374381+0000) 2024-02-20T09:31:08.450 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:08.377+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67157 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:08.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:08.373+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.374442+0000) 2024-02-20T09:31:08.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:08.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:08.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:08.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:08.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:08.196+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:08.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:08.197+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:08.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:08.253+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.254885+0000) 2024-02-20T09:31:08.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:08.373+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.374146+0000) 2024-02-20T09:31:08.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:08.562+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.563394+0000) 2024-02-20T09:31:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:08.549+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.550123+0000) 2024-02-20T09:31:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:08.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:08.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:08.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:08.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:08.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:09.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:09.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:09.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:09.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:09.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:09.280+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:09.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:09.281+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:09.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:09.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:09.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:09.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:09.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:09.159+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:09.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:09.160+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:09.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:09.254+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:09.255671+0000) 2024-02-20T09:31:09.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:09.565+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:09.566364+0000) 2024-02-20T09:31:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:09.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:09.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:09.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:09.516+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:09.552+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:09.553309+0000) 2024-02-20T09:31:10.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:10.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:10.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:10.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:10.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:10.249+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:10.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:10.250+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:10.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:10.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:10.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:10.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:10.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:10.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:10.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:10.152+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:10.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:10.258+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:10.259246+0000) 2024-02-20T09:31:10.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:10.568+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:10.569745+0000) 2024-02-20T09:31:10.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:10.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:10.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:10.553+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:10.554+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:10.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:10.554+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:10.555926+0000) 2024-02-20T09:31:11.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:11.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:11.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:11.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:11.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:11.292+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:11.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:11.292+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:11.450 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:11.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:11.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:11.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:11.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:11.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:11.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:11.135+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:11.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:11.260+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:11.261927+0000) 2024-02-20T09:31:11.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:11.572+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:11.573369+0000) 2024-02-20T09:31:11.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:11.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:11.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:11.504+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:11.505+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:11.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:11.557+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:11.558148+0000) 2024-02-20T09:31:12.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:12.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:12.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:12.143+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:12.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:12.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:12.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:12.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:12.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:12.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:12.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:12.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:12.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:12.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:12.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:12.263+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.264234+0000) 2024-02-20T09:31:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:12.574+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.575665+0000) 2024-02-20T09:31:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:12.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:12.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:12.547+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:12.548+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:12.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:12.560+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.561062+0000) 2024-02-20T09:31:13.366 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:13.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:13.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:13.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:13.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:13.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:13.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:13.272+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:13.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:13.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:13.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:13.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:13.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:13.110+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:13.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:13.111+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:13.368 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:13.265+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.266741+0000) 2024-02-20T09:31:13.698 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:13.385+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67195 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:13.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:13.382+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.383691+0000) 2024-02-20T09:31:13.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:13.382+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.383234+0000) 2024-02-20T09:31:13.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:13.575+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.576919+0000) 2024-02-20T09:31:13.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:13.382+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.383382+0000) 2024-02-20T09:31:13.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:13.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:13.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:13.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:13.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:13.542+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:13.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:13.542+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:13.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:13.560+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.561855+0000) 2024-02-20T09:31:14.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:14.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:14.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:14.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:14.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:14.272+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:14.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:14.273+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:14.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:14.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:14.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:14.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:14.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:14.151+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:14.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:14.152+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:14.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:14.266+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:14.267621+0000) 2024-02-20T09:31:14.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:14.578+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:14.579342+0000) 2024-02-20T09:31:14.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:14.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:14.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:14.557+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:14.558+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:14.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:14.563+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:14.564347+0000) 2024-02-20T09:31:15.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:15.201+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:15.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:15.201+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:15.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:15.201+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:15.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:15.202+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:15.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:15.269+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:15.270131+0000) 2024-02-20T09:31:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:15.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:15.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:15.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:15.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:15.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:15.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:15.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:15.581+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:15.582077+0000) 2024-02-20T09:31:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:15.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:15.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:15.540+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:15.541+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:15.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:15.566+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:15.566952+0000) 2024-02-20T09:31:16.507 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:16.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:16.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:16.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:16.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:16.265+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:16.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:16.266+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:16.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:16.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:16.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:16.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:16.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:16.231+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:16.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:16.232+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:16.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:16.271+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:16.272330+0000) 2024-02-20T09:31:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:16.584+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:16.585643+0000) 2024-02-20T09:31:16.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:16.563+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:16.563+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:16.563+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:16.564+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:16.569+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:16.570578+0000) 2024-02-20T09:31:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:17.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:17.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:17.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:17.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:17.309+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:17.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:17.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:17.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:17.210+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:17.211+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:17.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:17.274+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.274957+0000) 2024-02-20T09:31:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:17.587+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.588140+0000) 2024-02-20T09:31:17.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:17.535+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:17.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:17.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:17.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:17.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:17.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:17.536+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:17.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:17.573+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.574197+0000) 2024-02-20T09:31:18.507 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:18.393+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67236 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:18.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:18.390+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.390982+0000) 2024-02-20T09:31:18.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:18.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:18.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:18.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:18.334+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:18.335+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:18.391+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.392567+0000) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:18.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:18.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:18.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:18.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:18.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:18.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:18.276+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.277379+0000) 2024-02-20T09:31:18.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:18.389+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.390591+0000) 2024-02-20T09:31:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:18.588+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.589125+0000) 2024-02-20T09:31:18.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:18.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:18.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:18.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:18.570+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:18.574+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.575355+0000) 2024-02-20T09:31:19.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:19.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:19.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:19.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:19.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:19.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:19.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:19.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:19.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:19.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:19.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:19.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:19.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:19.221+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:19.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:19.222+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:19.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:19.277+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:19.278216+0000) 2024-02-20T09:31:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:19.590+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:19.591634+0000) 2024-02-20T09:31:19.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:19.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:19.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:19.536+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:19.537+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:19.577+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:19.577964+0000) 2024-02-20T09:31:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:20.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:20.320+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:20.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:20.321+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:20.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:20.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:20.205+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:20.206+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:20.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:20.279+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:20.280711+0000) 2024-02-20T09:31:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:20.592+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:20.593746+0000) 2024-02-20T09:31:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:20.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:20.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:20.569+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:20.570+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:20.579+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:20.580634+0000) 2024-02-20T09:31:21.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:21.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:21.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:21.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:21.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:21.369+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:21.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:21.370+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:21.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:21.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:21.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:21.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:21.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:21.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:21.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:21.254+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:21.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:21.282+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:21.283144+0000) 2024-02-20T09:31:21.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:21.595+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:21.595971+0000) 2024-02-20T09:31:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:21.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:21.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:21.523+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:21.524+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:21.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:21.583+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:21.584261+0000) 2024-02-20T09:31:22.556 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:22.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:22.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:22.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:22.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:22.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:22.557 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:22.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:22.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:22.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:22.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:22.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:22.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:22.282+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:22.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:22.283+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:22.558 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:22.284+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.285282+0000) 2024-02-20T09:31:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:22.597+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.598579+0000) 2024-02-20T09:31:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:22.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:22.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:22.555+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:22.555+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:22.585+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.586897+0000) 2024-02-20T09:31:23.570 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:23.401+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67274 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:23.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:23.397+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.398682+0000) 2024-02-20T09:31:23.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:23.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:23.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:23.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:23.571 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:23.568+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:23.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:23.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:23.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:23.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:23.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:23.302+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:23.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:23.303+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:23.572 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:23.398+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.399631+0000) 2024-02-20T09:31:23.572 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:23.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:23.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:23.275+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:23.276+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:23.287+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.288622+0000) 2024-02-20T09:31:23.573 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:23.398+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.399544+0000) 2024-02-20T09:31:23.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:23.598+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.599519+0000) 2024-02-20T09:31:23.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:23.569+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:23.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:23.586+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.587712+0000) 2024-02-20T09:31:24.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:24.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:24.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:24.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:24.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:24.286+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:24.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:24.287+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:24.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:24.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:24.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:24.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:24.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:24.280+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:24.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:24.281+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:24.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:24.288+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:24.289538+0000) 2024-02-20T09:31:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:24.602+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:24.603028+0000) 2024-02-20T09:31:24.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:24.589+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:24.590292+0000) 2024-02-20T09:31:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:24.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:24.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:24.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:24.601+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:25.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:25.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:25.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:25.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:25.239+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:25.240+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:25.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:25.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:25.246+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:25.247+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:25.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:25.290+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:25.291582+0000) 2024-02-20T09:31:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:25.604+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:25.605531+0000) 2024-02-20T09:31:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:25.592+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:25.593873+0000) 2024-02-20T09:31:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:25.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:25.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:25.609+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:25.610+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:26.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:26.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:26.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:26.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:26.261+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:26.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:26.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:26.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:26.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:26.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:26.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:26.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:26.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:26.293+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:26.294043+0000) 2024-02-20T09:31:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:26.606+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:26.607820+0000) 2024-02-20T09:31:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:26.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:26.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:26.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:26.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:26.563+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:26.595+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:26.596446+0000) 2024-02-20T09:31:27.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:27.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:27.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:27.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:27.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:27.232+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:27.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:27.233+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:27.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:27.295+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.296778+0000) 2024-02-20T09:31:27.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:27.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:27.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:27.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:27.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:27.309+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:27.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:27.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:27.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:27.609+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.610059+0000) 2024-02-20T09:31:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:27.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:27.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:27.574+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:27.575+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:27.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:27.599+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.600057+0000) 2024-02-20T09:31:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:28.406+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.407059+0000) 2024-02-20T09:31:28.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:28.410+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67315 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:28.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:28.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:28.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:28.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:28.406+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.407301+0000) 2024-02-20T09:31:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:28.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:28.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:28.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:28.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:28.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:28.298+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.299285+0000) 2024-02-20T09:31:28.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:28.406+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.407340+0000) 2024-02-20T09:31:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:28.609+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.610800+0000) 2024-02-20T09:31:28.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:28.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:28.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:28.533+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:28.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:28.600+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.601353+0000) 2024-02-20T09:31:29.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:29.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:29.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:29.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:29.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:29.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:29.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:29.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:29.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:29.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:29.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:29.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:29.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:29.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:29.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:29.299+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:29.300112+0000) 2024-02-20T09:31:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:29.613+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:29.614135+0000) 2024-02-20T09:31:29.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:29.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:29.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:29.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:29.563+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:29.602+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:29.603866+0000) 2024-02-20T09:31:30.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:30.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:30.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:30.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:30.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:30.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:30.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:30.149+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:30.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:30.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:30.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:30.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:30.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:30.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:30.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:30.278+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:30.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:30.301+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:30.302681+0000) 2024-02-20T09:31:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:30.616+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:30.617668+0000) 2024-02-20T09:31:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:30.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:30.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:30.519+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:30.520+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:30.604+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:30.605525+0000) 2024-02-20T09:31:31.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:31.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:31.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:31.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:31.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:31.109+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:31.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:31.110+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:31.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:31.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:31.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:31.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:31.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:31.267+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:31.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:31.268+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:31.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:31.305+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:31.306303+0000) 2024-02-20T09:31:31.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:31.619+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:31.620258+0000) 2024-02-20T09:31:31.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:31.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:31.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:31.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:31.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:31.514+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:31.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:31.515+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:31.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:31.607+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:31.608217+0000) 2024-02-20T09:31:32.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:32.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:32.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:32.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:32.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:32.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:32.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:32.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:32.611 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:32.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:32.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:32.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:32.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:32.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:32.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:32.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:32.612 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:32.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:32.613 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:32.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:32.613 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:32.271+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:32.613 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:32.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:32.613 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:32.309+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.309968+0000) 2024-02-20T09:31:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:32.609+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.610777+0000) 2024-02-20T09:31:32.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:32.621+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.622793+0000) 2024-02-20T09:31:33.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:33.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:33.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:33.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:33.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:33.103+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:33.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:33.104+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:33.378 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:33.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:33.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:33.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:33.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:33.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:33.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:33.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:33.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:33.311+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.312402+0000) 2024-02-20T09:31:33.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:33.415+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.416619+0000) 2024-02-20T09:31:33.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:33.622+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.623789+0000) 2024-02-20T09:31:33.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:33.413+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.414906+0000) 2024-02-20T09:31:33.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:33.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:33.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:33.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:33.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:33.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:33.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:33.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:33.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:33.610+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.611847+0000) 2024-02-20T09:31:33.703 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:33.418+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67353 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:33.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:33.414+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.415477+0000) 2024-02-20T09:31:34.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:34.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:34.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:34.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:34.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:34.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:34.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:34.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:34.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:34.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:34.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:34.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:34.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:34.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:34.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:34.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:34.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:34.312+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:34.313354+0000) 2024-02-20T09:31:34.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:34.625+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:34.625950+0000) 2024-02-20T09:31:34.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:34.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:34.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:34.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:34.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:34.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:34.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:34.485+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:34.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:34.612+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:34.613768+0000) 2024-02-20T09:31:35.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:35.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:35.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:35.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:35.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:35.100+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:35.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:35.101+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:35.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:35.314+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:35.315889+0000) 2024-02-20T09:31:35.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:35.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:35.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:35.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:35.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:35.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:35.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:35.343+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:35.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:35.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:35.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:35.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:35.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:35.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:35.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:35.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:35.631 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:35.614+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:35.616350+0000) 2024-02-20T09:31:35.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:35.627+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:35.629297+0000) 2024-02-20T09:31:36.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:36.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:36.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:36.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:36.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:36.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:36.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:36.128+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:36.317+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:36.319446+0000) 2024-02-20T09:31:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:36.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:36.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:36.386 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:36.382+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:36.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:36.630+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:36.632166+0000) 2024-02-20T09:31:36.701 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:36.383+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:36.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:36.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:36.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:36.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:36.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:36.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:36.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:36.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:36.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:36.616+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:36.618795+0000) 2024-02-20T09:31:37.394 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:37.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:37.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:37.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:37.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:37.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:37.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:37.138+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:37.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:37.320+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.322051+0000) 2024-02-20T09:31:37.395 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:37.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:37.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:37.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:37.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:37.358+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:37.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:37.359+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:37.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:37.633+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.635488+0000) 2024-02-20T09:31:37.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:37.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:37.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:37.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:37.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:37.450+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:37.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:37.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:37.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:37.619+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.621446+0000) 2024-02-20T09:31:38.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:38.428+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67394 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:38.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:38.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:38.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:38.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:38.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:38.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:38.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:38.182+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:38.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:38.422+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.423967+0000) 2024-02-20T09:31:38.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:38.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:38.438 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:38.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:38.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:38.418+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:38.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:38.419+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:38.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:38.422+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.424524+0000) 2024-02-20T09:31:38.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:38.322+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.324285+0000) 2024-02-20T09:31:38.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:38.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:38.439 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:38.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:38.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:38.379+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:38.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:38.380+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:38.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:38.421+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.423059+0000) 2024-02-20T09:31:38.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:38.635+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.636478+0000) 2024-02-20T09:31:38.701 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:38.620+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.622341+0000) 2024-02-20T09:31:39.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:39.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:39.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:39.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:39.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:39.161+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:39.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:39.162+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:39.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:39.324+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:39.325122+0000) 2024-02-20T09:31:39.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:39.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:39.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:39.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:39.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:39.374+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:39.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:39.375+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:39.638+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:39.639350+0000) 2024-02-20T09:31:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:39.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:39.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:39.468+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:39.469+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:39.624+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:39.625091+0000) 2024-02-20T09:31:40.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:40.326+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:40.327596+0000) 2024-02-20T09:31:40.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:40.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:40.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:40.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:40.451 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:40.337+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:40.452 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:40.338+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:40.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:40.170+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:40.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:40.170+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:40.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:40.170+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:40.452 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:40.171+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:40.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:40.640+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:40.641859+0000) 2024-02-20T09:31:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:40.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:40.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:40.494+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:40.495+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:40.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:40.626+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:40.627653+0000) 2024-02-20T09:31:41.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:41.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:41.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:41.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:41.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:41.180+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:41.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:41.181+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:41.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:41.329+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:41.330076+0000) 2024-02-20T09:31:41.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:41.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:41.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:41.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:41.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:41.366+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:41.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:41.367+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:41.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:41.644+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:41.645503+0000) 2024-02-20T09:31:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:41.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:41.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:41.483+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:41.484+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:41.629+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:41.630165+0000) 2024-02-20T09:31:42.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:42.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:42.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:42.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:42.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:42.185+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:42.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:42.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:42.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:42.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:42.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:42.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:42.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:42.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:42.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:42.478+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:42.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:42.331+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.332525+0000) 2024-02-20T09:31:42.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:42.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:42.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:42.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:42.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:42.405+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:42.636 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:42.406+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:42.647+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.648181+0000) 2024-02-20T09:31:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:42.633+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.634050+0000) 2024-02-20T09:31:43.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:43.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:43.428 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:43.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:43.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:43.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:43.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:43.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:43.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:43.334+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.335031+0000) 2024-02-20T09:31:43.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:43.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:43.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:43.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:43.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:43.426+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:43.702 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:43.437+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67432 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:43.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:43.435+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.436055+0000) 2024-02-20T09:31:43.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:43.648+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.649360+0000) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:43.427+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:43.432+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.433919+0000) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:43.434+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.435130+0000) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:43.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:43.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:43.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:43.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:43.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:43.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:43.634+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.635009+0000) 2024-02-20T09:31:44.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:44.335+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:44.336352+0000) 2024-02-20T09:31:44.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:44.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:44.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:44.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:44.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:44.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:44.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:44.126+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:44.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:44.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:44.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:44.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:44.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:44.472+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:44.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:44.473+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:44.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:44.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:44.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:44.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:44.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:44.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:44.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:44.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:44.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:44.651+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:44.652078+0000) 2024-02-20T09:31:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:44.636+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:44.637382+0000) 2024-02-20T09:31:45.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:45.338+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:45.339668+0000) 2024-02-20T09:31:45.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:45.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:45.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:45.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:45.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:45.131+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:45.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:45.132+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:45.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:45.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:45.655 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:45.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:45.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:45.500+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:45.656 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:45.501+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:45.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:45.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:45.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:45.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:45.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:45.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:45.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:45.391+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:45.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:45.639+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:45.640224+0000) 2024-02-20T09:31:45.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:45.653+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:45.654509+0000) 2024-02-20T09:31:46.367 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:46.342+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:46.342972+0000) 2024-02-20T09:31:46.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:46.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:46.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:46.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:46.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:46.112+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:46.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:46.113+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:46.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:46.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:46.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:46.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:46.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:46.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:46.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:46.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:46.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:46.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:46.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:46.522+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:46.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:46.523+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:46.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:46.524+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:46.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:46.656+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:46.657013+0000) 2024-02-20T09:31:46.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:46.642+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:46.643582+0000) 2024-02-20T09:31:47.379 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:47.344+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.345647+0000) 2024-02-20T09:31:47.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:47.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:47.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:47.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:47.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:47.098+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:47.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:47.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:47.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:47.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:47.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:47.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:47.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:47.394+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:47.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:47.395+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:47.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:47.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:47.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:47.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:47.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:47.482+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:47.649 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:47.483+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:47.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:47.658+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.659109+0000) 2024-02-20T09:31:47.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:47.646+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.647058+0000) 2024-02-20T09:31:48.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:48.346+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.347809+0000) 2024-02-20T09:31:48.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:48.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:48.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:48.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:48.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:48.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:48.405 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:48.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:48.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:48.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:48.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:48.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:48.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:48.373+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:48.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:48.374+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:48.661 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:48.446+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67472 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:48.661 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:48.443+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.443961+0000) 2024-02-20T09:31:48.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:48.446+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.447074+0000) 2024-02-20T09:31:48.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:48.647+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.648124+0000) 2024-02-20T09:31:48.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:48.443+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.444257+0000) 2024-02-20T09:31:48.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:48.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:48.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:48.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:48.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:48.527+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:48.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:48.528+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:48.659+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.660446+0000) 2024-02-20T09:31:49.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:49.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:49.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:49.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:49.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:49.181+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:49.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:49.182+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:49.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:49.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:49.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:49.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:49.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:49.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:49.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:49.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:49.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:49.347+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:49.348800+0000) 2024-02-20T09:31:49.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:49.662+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:49.663476+0000) 2024-02-20T09:31:49.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:49.649+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:49.650563+0000) 2024-02-20T09:31:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:49.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:49.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:49.558+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:49.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:49.559+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:50.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:50.351+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:50.352377+0000) 2024-02-20T09:31:50.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:50.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:50.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:50.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:50.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:50.226+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:50.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:50.227+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:50.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:50.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:50.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:50.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:50.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:50.332+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:50.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:50.333+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:50.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:50.652+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:50.653059+0000) 2024-02-20T09:31:50.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:50.665+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:50.665974+0000) 2024-02-20T09:31:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:50.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:50.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:50.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:50.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:50.546+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:51.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:51.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:51.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:51.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:51.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:51.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:51.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:51.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:51.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:51.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:51.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:51.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:51.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:51.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:51.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:51.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:51.456 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:51.353+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:51.354845+0000) 2024-02-20T09:31:51.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:51.668+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:51.668926+0000) 2024-02-20T09:31:51.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:51.654+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:51.655869+0000) 2024-02-20T09:31:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:51.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:51.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:51.545+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:51.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:51.547+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:52.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:52.356+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.357463+0000) 2024-02-20T09:31:52.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:52.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:52.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:52.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:52.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:52.236+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:52.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:52.237+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:52.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:52.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:52.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:52.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:52.658+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.659452+0000) 2024-02-20T09:31:52.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:52.670+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.671636+0000) 2024-02-20T09:31:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:52.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:52.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:52.592+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:52.593+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:53.575 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:53.454+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67511 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:53.575 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:53.359+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.360117+0000) 2024-02-20T09:31:53.576 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:53.450+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.451377+0000) 2024-02-20T09:31:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:53.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:53.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:53.286+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:53.287+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:53.450+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.451831+0000) 2024-02-20T09:31:53.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:53.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:53.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:53.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:53.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:53.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:53.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:53.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:53.577 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:53.450+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.451285+0000) 2024-02-20T09:31:53.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:53.671+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.672577+0000) 2024-02-20T09:31:53.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:53.659+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.660484+0000) 2024-02-20T09:31:53.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:53.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:53.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:53.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:53.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:53.635+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:53.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:53.636+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:54.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:54.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:54.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:54.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:54.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:54.325+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:54.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:54.325+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:54.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:54.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:54.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:54.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:54.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:54.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:54.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:54.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:54.635 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:54.360+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:54.360944+0000) 2024-02-20T09:31:54.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:54.675+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:54.676082+0000) 2024-02-20T09:31:54.943 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:54.661+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:54.662773+0000) 2024-02-20T09:31:54.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:54.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:54.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:54.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:54.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:54.631+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:54.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:54.632+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:55.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:55.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:55.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:55.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:55.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:55.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:55.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:55.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:55.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:55.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:55.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:55.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:55.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:55.278+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:55.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:55.279+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:55.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:55.362+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:55.362951+0000) 2024-02-20T09:31:55.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:55.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:55.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:55.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:55.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:55.633+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:55.677+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:55.678097+0000) 2024-02-20T09:31:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:55.663+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:55.664898+0000) 2024-02-20T09:31:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:55.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:56.627 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:56.364+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:56.365209+0000) 2024-02-20T09:31:56.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:56.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:56.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:56.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:56.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:56.329+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:56.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:56.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:56.628 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:56.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:56.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:56.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:56.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:56.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:56.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:56.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:56.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:56.680+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:56.681159+0000) 2024-02-20T09:31:56.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:56.666+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:56.667883+0000) 2024-02-20T09:31:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:56.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:56.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:56.625+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:56.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:56.626+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:57.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:57.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:57.604 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:57.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:57.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:57.298+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:57.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:57.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:57.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:57.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:57.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:57.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:57.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:57.364+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:57.605 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:57.365+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:57.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:57.367+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.368801+0000) 2024-02-20T09:31:57.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:57.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:57.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:57.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:57.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:57.602+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:57.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:57.670+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.671244+0000) 2024-02-20T09:31:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:57.681+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.682822+0000) 2024-02-20T09:31:57.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:57.603+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:58.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:58.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:58.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:58.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:58.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:58.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:58.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:58.459+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.460009+0000) 2024-02-20T09:31:58.674 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:31:58.462+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67551 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:31:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:58.371+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.372082+0000) 2024-02-20T09:31:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:58.460+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.461926+0000) 2024-02-20T09:31:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:58.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:58.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:58.556+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:58.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:58.557+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:58.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:58.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:58.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:58.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:58.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:58.405+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:58.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:58.406+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:58.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:58.459+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.460264+0000) 2024-02-20T09:31:58.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:58.683+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.684183+0000) 2024-02-20T09:31:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:58.671+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.672586+0000) 2024-02-20T09:31:59.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:59.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:59.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:59.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:59.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:59.300+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:31:59.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:59.300+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:59.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:59.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:59.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:59.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:59.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:59.395+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:31:59.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:59.396+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:59.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:59.372+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:59.373456+0000) 2024-02-20T09:31:59.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:59.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:59.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:59.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:59.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:59.540+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:31:59.679 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:31:59.541+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:31:59.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:31:59.685+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:59.686310+0000) 2024-02-20T09:31:59.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:31:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:31:59.675+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:59.676261+0000) 2024-02-20T09:32:00.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:00.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:00.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:00.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:00.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:00.330+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:00.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:00.330+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:00.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:00.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:00.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:00.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:00.375+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:00.375951+0000) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:00.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:00.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:00.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:00.564+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:00.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:00.564+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:00.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:00.688+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:00.688940+0000) 2024-02-20T09:32:00.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:00.677+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:00.678924+0000) 2024-02-20T09:32:01.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:01.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:01.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:01.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:01.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:01.307+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:01.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:01.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:01.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:01.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:01.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:01.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:01.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:01.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:01.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:01.453+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:01.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:01.377+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:01.378649+0000) 2024-02-20T09:32:01.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:01.550+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:01.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:01.550+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:01.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:01.550+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:01.684 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:01.550+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:01.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:01.680+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:01.681795+0000) 2024-02-20T09:32:01.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:01.690+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:01.691036+0000) 2024-02-20T09:32:02.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:02.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:02.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:02.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:02.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:02.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:02.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:02.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:02.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:02.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:02.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:02.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:02.437+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:02.438+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:02.380+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.381163+0000) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:02.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:02.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:02.600+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:02.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:02.602+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:02.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:02.692+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.693452+0000) 2024-02-20T09:32:02.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:02.684+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.685051+0000) 2024-02-20T09:32:03.592 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:03.470+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67590 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:03.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:03.382+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.383256+0000) 2024-02-20T09:32:03.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:03.467+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.468150+0000) 2024-02-20T09:32:03.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:03.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:03.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:03.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:03.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:03.340+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:03.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:03.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:03.593 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:03.466+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.467599+0000) 2024-02-20T09:32:03.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:03.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:03.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:03.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:03.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:03.402+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:03.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:03.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:03.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:03.466+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.467904+0000) 2024-02-20T09:32:03.877 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:03.693+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.694498+0000) 2024-02-20T09:32:03.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:03.685+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.686071+0000) 2024-02-20T09:32:03.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:03.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:03.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:03.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:03.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:03.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:03.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:03.627+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:04.420 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:04.383+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:04.384339+0000) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:04.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:04.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:04.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:04.367+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:04.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:04.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:04.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:04.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:04.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:04.308+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:04.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:04.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:04.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:04.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:04.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:04.580+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:04.690 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:04.580+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:04.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:04.696+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:04.696943+0000) 2024-02-20T09:32:04.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:04.688+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:04.689280+0000) 2024-02-20T09:32:05.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:05.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:05.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:05.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:05.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:05.356+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:05.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:05.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:05.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:05.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:05.620 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:05.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:05.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:05.297+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:05.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:05.298+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:05.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:05.387+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:05.387960+0000) 2024-02-20T09:32:05.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:05.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:05.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:05.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:05.621 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:05.617+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:05.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:05.699+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:05.700316+0000) 2024-02-20T09:32:05.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:05.690+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:05.691849+0000) 2024-02-20T09:32:05.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:05.618+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:06.590 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:06.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:06.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:06.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:06.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:06.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:06.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:06.267+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:06.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:06.268+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:06.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:06.389+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:06.389968+0000) 2024-02-20T09:32:06.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:06.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:06.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:06.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:06.592 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:06.588+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:06.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:06.702+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:06.703657+0000) 2024-02-20T09:32:06.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:06.693+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:06.694406+0000) 2024-02-20T09:32:06.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:06.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:07.512 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:07.391+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.392165+0000) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:07.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:07.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:07.351+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:07.352+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:07.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:07.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:07.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:07.235+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:07.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:07.236+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:07.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:07.706+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.707057+0000) 2024-02-20T09:32:07.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:07.695+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.696470+0000) 2024-02-20T09:32:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:07.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:07.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:07.606+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:07.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:07.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:08.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:08.479+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67630 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:08.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:08.393+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.394805+0000) 2024-02-20T09:32:08.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:08.475+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.476760+0000) 2024-02-20T09:32:08.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:08.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:08.601 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:08.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:08.283+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:08.285+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:08.476+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.477447+0000) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:08.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:08.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:08.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:08.602 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:08.357+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:08.603 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:08.475+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.476476+0000) 2024-02-20T09:32:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:08.707+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.708478+0000) 2024-02-20T09:32:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:08.696+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.697565+0000) 2024-02-20T09:32:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:08.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:08.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:08.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:08.634+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:09.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:09.394+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:09.395782+0000) 2024-02-20T09:32:09.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:09.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:09.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:09.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:09.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:09.314+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:09.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:09.315+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:09.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:09.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:09.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:09.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:09.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:09.311+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:09.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:09.312+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:09.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:09.709+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:09.710486+0000) 2024-02-20T09:32:09.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:09.698+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:09.699531+0000) 2024-02-20T09:32:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:09.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:09.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:09.605+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:09.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:09.606+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:10.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:10.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:10.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:10.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:10.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:10.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:10.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:10.299+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:10.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:10.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:10.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:10.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:10.334+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:10.397+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:10.398383+0000) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:10.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:10.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:10.614+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:10.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:10.711+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:10.712600+0000) 2024-02-20T09:32:10.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:10.700+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:10.701689+0000) 2024-02-20T09:32:10.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:10.615+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:11.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:11.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:11.605 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:11.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:11.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:11.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:11.606 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:11.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:11.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:11.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:11.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:11.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:11.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:11.317+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:11.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:11.318+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:11.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:11.399+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:11.400472+0000) 2024-02-20T09:32:11.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:11.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:11.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:11.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:11.607 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:11.603+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:11.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:11.713+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:11.714867+0000) 2024-02-20T09:32:11.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:11.604+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:11.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:11.703+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:11.703951+0000) 2024-02-20T09:32:12.628 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:12.402+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.403309+0000) 2024-02-20T09:32:12.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:12.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:12.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:12.352+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:12.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:12.353+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:12.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:12.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:12.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:12.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:12.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:12.358+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:12.630 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:12.359+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:12.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:12.716+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.717423+0000) 2024-02-20T09:32:12.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:12.706+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.707541+0000) 2024-02-20T09:32:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:12.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:12.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:12.627+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:12.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:12.627+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:13.665 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:13.488+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67669 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:13.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:13.404+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.405789+0000) 2024-02-20T09:32:13.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:13.484+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.485397+0000) 2024-02-20T09:32:13.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:13.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:13.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:13.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:13.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:13.401+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:13.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:13.402+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:13.667 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:13.484+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.485249+0000) 2024-02-20T09:32:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:13.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:13.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:13.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:13.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:13.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:13.484+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.485277+0000) 2024-02-20T09:32:13.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:13.717+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.718516+0000) 2024-02-20T09:32:13.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:13.708+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.709017+0000) 2024-02-20T09:32:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:13.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:13.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:13.663+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:13.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:13.664+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:14.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:14.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:14.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:14.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:14.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:14.375+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:14.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:14.376+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:14.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:14.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:14.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:14.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:14.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:14.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:14.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:14.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:14.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:14.406+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:14.407305+0000) 2024-02-20T09:32:14.700 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:14.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:14.700 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:14.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:14.700 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:14.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:14.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:14.720+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:14.721149+0000) 2024-02-20T09:32:14.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:14.711+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:14.712602+0000) 2024-02-20T09:32:14.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:14.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:15.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:15.408+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:15.409251+0000) 2024-02-20T09:32:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:15.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:15.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:15.333+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:15.334+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:15.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:15.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:15.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:15.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:15.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:15.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:15.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:15.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:15.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:15.722+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:15.723343+0000) 2024-02-20T09:32:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:15.713+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:15.714738+0000) 2024-02-20T09:32:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:15.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:15.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:15.732+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:15.733+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:16.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:16.411+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:16.412574+0000) 2024-02-20T09:32:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:16.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:16.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:16.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:16.316+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:16.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:16.318+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:16.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:16.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:16.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:16.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:16.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:16.386+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:16.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:16.388+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:16.725+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:16.726677+0000) 2024-02-20T09:32:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:16.717+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:16.718069+0000) 2024-02-20T09:32:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:16.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:16.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:16.752+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:16.754+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:17.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:17.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:17.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:17.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:17.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:17.348+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:17.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:17.349+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:17.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:17.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:17.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:17.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:17.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:17.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:17.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:17.369+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:17.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:17.414+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.415474+0000) 2024-02-20T09:32:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:17.727+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.728874+0000) 2024-02-20T09:32:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:17.720+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.721420+0000) 2024-02-20T09:32:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:17.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:17.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:17.778+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:17.779+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:18.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:18.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:18.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:18.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:18.388+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:18.389+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:18.492+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.493317+0000) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:18.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:18.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:18.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:18.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:18.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:18.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:18.492+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.493385+0000) 2024-02-20T09:32:18.707 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:18.497+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67709 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:18.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:18.417+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.418846+0000) 2024-02-20T09:32:18.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:18.494+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.495483+0000) 2024-02-20T09:32:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:18.729+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.730393+0000) 2024-02-20T09:32:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:18.721+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.722514+0000) 2024-02-20T09:32:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:18.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:18.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:18.803+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:18.804+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:19.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:19.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:19.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:19.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:19.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:19.371+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:19.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:19.372+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:19.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:19.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:19.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:19.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:19.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:19.423+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:19.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:19.424+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:19.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:19.419+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:19.420252+0000) 2024-02-20T09:32:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:19.732+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:19.733026+0000) 2024-02-20T09:32:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:19.725+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:19.726084+0000) 2024-02-20T09:32:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:19.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:19.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:19.788+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:19.789+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:20.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:20.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:20.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:20.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:20.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:20.350+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:20.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:20.351+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:20.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:20.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:20.706 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:20.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:20.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:20.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:20.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:20.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:20.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:20.422+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:20.423791+0000) 2024-02-20T09:32:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:20.734+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:20.735468+0000) 2024-02-20T09:32:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:20.727+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:20.728687+0000) 2024-02-20T09:32:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:20.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:20.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:20.817+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:20.818+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:21.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:21.425+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:21.426408+0000) 2024-02-20T09:32:21.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:21.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:21.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:21.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:21.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:21.346+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:21.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:21.347+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:21.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:21.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:21.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:21.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:21.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:21.414+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:21.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:21.415+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:21.736+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:21.737732+0000) 2024-02-20T09:32:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:21.730+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:21.731366+0000) 2024-02-20T09:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:21.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:21.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:21.861+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:21.862+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:22.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:22.427+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.428691+0000) 2024-02-20T09:32:22.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:22.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:22.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:22.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:22.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:22.357+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:22.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:22.358+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:22.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:22.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:22.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:22.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:22.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:22.381+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:22.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:22.382+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:22.740+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.741299+0000) 2024-02-20T09:32:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:22.732+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.733462+0000) 2024-02-20T09:32:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:22.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:22.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:22.880+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:22.881+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:23.627 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:23.505+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67748 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:23.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:23.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:23.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:23.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:23.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:23.347+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:23.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:23.348+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:23.628 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:23.501+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.502584+0000) 2024-02-20T09:32:23.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:23.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:23.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:23.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:23.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:23.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:23.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:23.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:23.629 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:23.503+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.504235+0000) 2024-02-20T09:32:23.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:23.430+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.431258+0000) 2024-02-20T09:32:23.630 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:23.502+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.503178+0000) 2024-02-20T09:32:23.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:23.741+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.742753+0000) 2024-02-20T09:32:23.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:23.733+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.734396+0000) 2024-02-20T09:32:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:23.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:23.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:24.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:23.884+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:24.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:23.885+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:24.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:24.431+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:24.432473+0000) 2024-02-20T09:32:24.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:24.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:24.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:24.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:24.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:24.315+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:24.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:24.316+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:24.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:24.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:24.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:24.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:24.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:24.357+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:24.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:24.358+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:24.745+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:24.746109+0000) 2024-02-20T09:32:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:24.735+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:24.736694+0000) 2024-02-20T09:32:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:24.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:24.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:24.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:24.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:25.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:25.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:25.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:25.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:25.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:25.282+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:25.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:25.283+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:25.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:25.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:25.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:25.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:25.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:25.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:25.664 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:25.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:25.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:25.434+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:25.435906+0000) 2024-02-20T09:32:25.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:25.739+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:25.740013+0000) 2024-02-20T09:32:25.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:25.748+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:25.749713+0000) 2024-02-20T09:32:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:25.883+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:25.883+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:25.883+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:25.884+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:26.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:26.437+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:26.438137+0000) 2024-02-20T09:32:26.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:26.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:26.514 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:26.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:26.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:26.315+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:26.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:26.316+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:26.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:26.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:26.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:26.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:26.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:26.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:26.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:26.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:26.837 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:26.752+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:26.753216+0000) 2024-02-20T09:32:26.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:26.741+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:26.742044+0000) 2024-02-20T09:32:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:26.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:27.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:26.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:26.835+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:27.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:26.836+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:27.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:27.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:27.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:27.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:27.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:27.222+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:27.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:27.223+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:27.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:27.439+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.440179+0000) 2024-02-20T09:32:27.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:27.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:27.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:27.278+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:27.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:27.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:27.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:27.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:28.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:27.755+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.756636+0000) 2024-02-20T09:32:28.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:27.743+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.744809+0000) 2024-02-20T09:32:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:27.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:27.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:27.802+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:28.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:27.803+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:28.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:28.441+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.442951+0000) 2024-02-20T09:32:28.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:28.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:28.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:28.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:28.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:28.269+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:28.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:28.270+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:28.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:28.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:28.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:28.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:28.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:28.256+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:28.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:28.257+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:28.780 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:28.513+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67788 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:28.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:28.509+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.511514+0000) 2024-02-20T09:32:28.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:28.756+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.758131+0000) 2024-02-20T09:32:28.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:28.509+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.511875+0000) 2024-02-20T09:32:28.782 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:28.743+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.745850+0000) 2024-02-20T09:32:28.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:28.510+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.512307+0000) 2024-02-20T09:32:28.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:28.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:28.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:28.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:28.782 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:28.777+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:29.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:28.779+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:29.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:29.441+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:29.443911+0000) 2024-02-20T09:32:29.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:29.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:29.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:29.225+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:29.226+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:29.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:29.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:29.297+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:29.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:29.297+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:30.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:29.758+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:29.760571+0000) 2024-02-20T09:32:30.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:29.747+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:29.749208+0000) 2024-02-20T09:32:30.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:29.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:30.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:29.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:30.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:29.787+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:30.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:29.787+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:30.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:30.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:30.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:30.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:30.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:30.178+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:30.447 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:30.179+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:30.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:30.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:30.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:30.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:30.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:30.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:30.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:30.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:30.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:30.443+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:30.445855+0000) 2024-02-20T09:32:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:30.760+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:30.762873+0000) 2024-02-20T09:32:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:30.749+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:30.751358+0000) 2024-02-20T09:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:30.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:30.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:30.766+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:30.766+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:31.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:31.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:31.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:31.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:31.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:31.371+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:31.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:31.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:31.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:31.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:31.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:31.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:31.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:31.193+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:31.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:31.194+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:31.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:31.447+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:31.449360+0000) 2024-02-20T09:32:31.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:31.763+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:31.764907+0000) 2024-02-20T09:32:31.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:31.752+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:31.753762+0000) 2024-02-20T09:32:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:31.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:31.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:31.737+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:31.738+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:32.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:32.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:32.367 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:32.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:32.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:32.198+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:32.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:32.199+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:32.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:32.451+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.452079+0000) 2024-02-20T09:32:32.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:32.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:32.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:32.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:32.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:32.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:32.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:32.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:32.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:32.755+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.756289+0000) 2024-02-20T09:32:32.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:32.766+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.767138+0000) 2024-02-20T09:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:32.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:32.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:32.742+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:32.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:32.743+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:33.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:33.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:33.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:33.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:33.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:33.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:33.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:33.175+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:33.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:33.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:33.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:33.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:33.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:33.399+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:33.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:33.401+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:33.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:33.519+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.520354+0000) 2024-02-20T09:32:33.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:33.518+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.519549+0000) 2024-02-20T09:32:33.707 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:33.522+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67827 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:33.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:33.453+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.454555+0000) 2024-02-20T09:32:33.707 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:33.518+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.519559+0000) 2024-02-20T09:32:33.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:33.767+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.768275+0000) 2024-02-20T09:32:33.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:33.756+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.757646+0000) 2024-02-20T09:32:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:33.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:33.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:33.736+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:33.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:33.737+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:34.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:34.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:34.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:34.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:34.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:34.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:34.435 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:34.139+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:34.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:34.454+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:34.455524+0000) 2024-02-20T09:32:34.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:34.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:34.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:34.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:34.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:34.432+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:34.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:34.434+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:34.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:34.769+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:34.770818+0000) 2024-02-20T09:32:34.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:34.759+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:34.760102+0000) 2024-02-20T09:32:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:34.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:34.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:34.751+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:34.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:34.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:35.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:35.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:35.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:35.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:35.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:35.127+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:35.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:35.128+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:35.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:35.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:35.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:35.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:35.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:35.404+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:35.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:35.405+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:35.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:35.457+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:35.458048+0000) 2024-02-20T09:32:35.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:35.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:35.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:35.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:35.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:35.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:35.762+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:35.763661+0000) 2024-02-20T09:32:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:35.772+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:35.773388+0000) 2024-02-20T09:32:35.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:35.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:36.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:36.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:36.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:36.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:36.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:36.144+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:36.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:36.145+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:36.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:36.365+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:36.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:36.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:36.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:36.366+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:36.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:36.366+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:36.459+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:36.460690+0000) 2024-02-20T09:32:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:36.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:36.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:36.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:36.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:36.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:36.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:37.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:36.775+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:36.775928+0000) 2024-02-20T09:32:37.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:36.765+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:36.766181+0000) 2024-02-20T09:32:37.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:37.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:37.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:37.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:37.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:37.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:37.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:37.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:37.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:37.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:37.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:37.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:37.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:37.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:37.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:37.157+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:37.462+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.463114+0000) 2024-02-20T09:32:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:37.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:37.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:37.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:37.662+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:37.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:37.663+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:38.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:37.777+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.778350+0000) 2024-02-20T09:32:38.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:37.768+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.769800+0000) 2024-02-20T09:32:38.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:38.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:38.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:38.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:38.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:38.379+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:38.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:38.379+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:38.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:38.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:38.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:38.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:38.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:38.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:38.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:38.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:38.527+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.527983+0000) 2024-02-20T09:32:38.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:38.531+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67867 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:38.526+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.527267+0000) 2024-02-20T09:32:38.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:38.465+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.466038+0000) 2024-02-20T09:32:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:38.527+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.528087+0000) 2024-02-20T09:32:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:38.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:38.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:38.619+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:38.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:38.620+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:39.120 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:38.769+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.770880+0000) 2024-02-20T09:32:39.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:38.778+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.779518+0000) 2024-02-20T09:32:39.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:39.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:39.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:39.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:39.121 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:39.118+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:39.384 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:39.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:39.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:39.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:39.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:39.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:39.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:39.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:39.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:39.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:39.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:39.466+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:39.466952+0000) 2024-02-20T09:32:39.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:39.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:39.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:39.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:39.658 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:39.656+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:39.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:39.781+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:39.782654+0000) 2024-02-20T09:32:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:39.772+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:39.773616+0000) 2024-02-20T09:32:39.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:39.657+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:40.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:40.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:40.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:40.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:40.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:40.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:40.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:40.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:40.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:40.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:40.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:40.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:40.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:40.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:40.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:40.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:40.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:40.468+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:40.469060+0000) 2024-02-20T09:32:40.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:40.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:40.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:40.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:40.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:40.608+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:40.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:40.609+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:40.784+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:40.785176+0000) 2024-02-20T09:32:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:40.775+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:40.776064+0000) 2024-02-20T09:32:41.324 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:41.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:41.324 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:41.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:41.324 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:41.054+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:41.324 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:41.054+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:41.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:41.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:41.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:41.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:41.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:41.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:41.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:41.323+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:41.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:41.471+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:41.472425+0000) 2024-02-20T09:32:41.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:41.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:41.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:41.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:41.641 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:41.638+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:41.786+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:41.787253+0000) 2024-02-20T09:32:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:41.778+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:41.779441+0000) 2024-02-20T09:32:41.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:41.639+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:42.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:42.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:42.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:42.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:42.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:42.089+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:42.340 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:42.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:42.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:42.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:42.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:42.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:42.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:42.337+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:42.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:42.338+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:42.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:42.474+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.474947+0000) 2024-02-20T09:32:42.637 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:42.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:42.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:42.788+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.789296+0000) 2024-02-20T09:32:42.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:42.780+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.781454+0000) 2024-02-20T09:32:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:42.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:42.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:42.634+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:42.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:42.635+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:43.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:43.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:43.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:43.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:43.092+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:43.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:43.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:43.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:43.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:43.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:43.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:43.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:43.300+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:43.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:43.539+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67906 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:43.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:43.535+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.536110+0000) 2024-02-20T09:32:43.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:43.535+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.536676+0000) 2024-02-20T09:32:43.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:43.476+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.477586+0000) 2024-02-20T09:32:43.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:43.534+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.535788+0000) 2024-02-20T09:32:43.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:43.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:43.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:43.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:43.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:43.589+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:43.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:43.590+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:43.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:43.789+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.790310+0000) 2024-02-20T09:32:43.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:43.782+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.783013+0000) 2024-02-20T09:32:44.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:44.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:44.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:44.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:44.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:44.284+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:44.385 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:44.285+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:44.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:44.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:44.125+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:44.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:44.125+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:44.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:44.477+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:44.478575+0000) 2024-02-20T09:32:44.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:44.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:44.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:44.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:44.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:44.637+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:44.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:44.784+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:44.785802+0000) 2024-02-20T09:32:44.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:44.638+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:44.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:44.791+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:44.792521+0000) 2024-02-20T09:32:45.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:45.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:45.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:45.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:45.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:45.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:45.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:45.242+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:45.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:45.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:45.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:45.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:45.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:45.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:45.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:45.151+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:45.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:45.479+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:45.480849+0000) 2024-02-20T09:32:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:45.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:45.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:45.687+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:45.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:45.687+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:46.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:45.794+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:45.794932+0000) 2024-02-20T09:32:46.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:45.787+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:45.788236+0000) 2024-02-20T09:32:46.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:46.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:46.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:46.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:46.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:46.244+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:46.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:46.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:46.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:46.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:46.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:46.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:46.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:46.172+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:46.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:46.173+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:46.482+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:46.483401+0000) 2024-02-20T09:32:46.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:46.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:46.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:46.659+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:46.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:46.660+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:47.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:46.797+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:46.798498+0000) 2024-02-20T09:32:47.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:46.789+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:46.790389+0000) 2024-02-20T09:32:47.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:47.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:47.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:47.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:47.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:47.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:47.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:47.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:47.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:47.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:47.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:47.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:47.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:47.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:47.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:47.155+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:47.485+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.486720+0000) 2024-02-20T09:32:47.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:47.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:47.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:47.671+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:47.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:47.672+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:48.119 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:47.791+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.792816+0000) 2024-02-20T09:32:48.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:47.800+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.801881+0000) 2024-02-20T09:32:48.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:48.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:48.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:48.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:48.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:48.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:48.385 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:48.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:48.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:48.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:48.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:48.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:48.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:48.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:48.386 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:48.188+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:48.649 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:48.547+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67945 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:48.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:48.544+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.545062+0000) 2024-02-20T09:32:48.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:48.544+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.545118+0000) 2024-02-20T09:32:48.650 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:48.488+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.489182+0000) 2024-02-20T09:32:48.650 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:48.544+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.545406+0000) 2024-02-20T09:32:48.650 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:48.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:48.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:48.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:48.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:48.647+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:48.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:48.802+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.803521+0000) 2024-02-20T09:32:48.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:48.648+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:48.793+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.794197+0000) 2024-02-20T09:32:49.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:49.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:49.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:49.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:49.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:49.165+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:49.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:49.166+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:49.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:49.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:49.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:49.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:49.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:49.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:49.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:49.225+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:49.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:49.489+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:49.490428+0000) 2024-02-20T09:32:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:49.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:49.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:49.607+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:49.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:49.608+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:50.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:49.804+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:49.805551+0000) 2024-02-20T09:32:50.151 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:49.796+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:49.797527+0000) 2024-02-20T09:32:50.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:50.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:50.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:50.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:50.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:50.148+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:50.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:50.149+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:50.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:50.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:50.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:50.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:50.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:50.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:50.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:50.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:50.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:50.491+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:50.492792+0000) 2024-02-20T09:32:50.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:50.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:50.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:50.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:50.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:50.636+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:50.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:50.637+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:50.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:50.806+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:50.807711+0000) 2024-02-20T09:32:50.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:50.800+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:50.800952+0000) 2024-02-20T09:32:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:51.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:51.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:51.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:51.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:51.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:51.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:51.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:51.183+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:51.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:51.185+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:51.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:51.495+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:51.496095+0000) 2024-02-20T09:32:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:51.802+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:51.803522+0000) 2024-02-20T09:32:51.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:51.808+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:51.809897+0000) 2024-02-20T09:32:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:51.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:51.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:51.679+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:51.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:51.680+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:52.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:52.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:52.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:52.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:52.386 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:52.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:52.387 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:52.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:52.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:52.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:52.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:52.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:52.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:52.182+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:52.387 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:52.184+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:52.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:52.498+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.499449+0000) 2024-02-20T09:32:52.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:52.812+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.813280+0000) 2024-02-20T09:32:52.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:52.805+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.806083+0000) 2024-02-20T09:32:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:52.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:52.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:52.651+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:52.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:52.652+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:53.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:53.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:53.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:53.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:53.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:53.187+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:53.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:53.188+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:53.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:53.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:53.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:53.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:53.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:53.227+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:53.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:53.228+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:53.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:53.556+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 67982 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:53.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:53.552+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.553721+0000) 2024-02-20T09:32:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:53.552+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.553723+0000) 2024-02-20T09:32:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:53.501+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.502784+0000) 2024-02-20T09:32:53.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:53.553+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.554498+0000) 2024-02-20T09:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:53.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:53.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:53.653+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:53.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:53.654+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:54.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:53.813+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.814535+0000) 2024-02-20T09:32:54.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:53.806+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.807701+0000) 2024-02-20T09:32:54.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:54.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:54.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:54.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:54.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:54.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:54.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:54.220+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:54.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:54.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:54.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:54.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:54.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:54.189+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:54.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:54.190+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:54.503+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:54.504161+0000) 2024-02-20T09:32:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:54.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:54.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:54.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:54.682+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:54.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:54.683+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:55.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:54.816+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:54.817844+0000) 2024-02-20T09:32:55.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:54.808+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:54.809788+0000) 2024-02-20T09:32:55.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:55.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:55.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:55.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:55.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:55.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:55.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:55.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:55.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:55.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:55.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:55.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:55.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:55.194+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:55.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:55.195+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:55.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:55.505+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:55.506564+0000) 2024-02-20T09:32:55.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:55.819+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:55.820367+0000) 2024-02-20T09:32:55.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:55.811+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:55.812427+0000) 2024-02-20T09:32:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:55.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:55.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:55.712+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:55.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:55.713+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:56.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:56.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:56.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:56.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:56.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:56.184+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:56.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:56.185+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:56.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:56.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:56.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:56.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:56.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:56.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:56.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:56.238+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:56.508+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:56.508991+0000) 2024-02-20T09:32:56.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:56.699+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:56.699+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:56.699+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:56.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:56.699+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:57.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:56.814+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:56.815012+0000) 2024-02-20T09:32:57.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:56.822+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:56.822950+0000) 2024-02-20T09:32:57.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:57.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:57.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:57.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:57.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:57.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:57.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:57.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:57.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:57.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:57.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:57.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:57.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:57.818 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:57.510+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.511536+0000) 2024-02-20T09:32:57.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:57.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:57.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:57.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:57.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:57.723+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:57.819 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:57.724+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:58.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:57.824+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.825318+0000) 2024-02-20T09:32:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:57.816+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.817229+0000) 2024-02-20T09:32:58.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:58.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:58.480 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:58.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:58.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:58.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:58.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:58.225+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:58.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:58.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:58.240+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:58.241+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:58.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:32:58.564+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68023 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:32:58.820 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:58.560+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.561473+0000) 2024-02-20T09:32:58.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:58.561+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.562216+0000) 2024-02-20T09:32:58.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:58.513+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.513957+0000) 2024-02-20T09:32:58.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:58.563+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.564286+0000) 2024-02-20T09:32:58.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:58.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:58.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:58.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:58.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:58.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:58.821 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:58.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:59.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:58.817+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.818639+0000) 2024-02-20T09:32:59.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:58.825+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.826741+0000) 2024-02-20T09:32:59.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:59.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:59.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:59.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:59.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:59.179+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:32:59.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:59.180+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:32:59.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:59.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:59.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:59.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:59.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:59.221+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:32:59.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:59.222+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:59.513+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:59.514924+0000) 2024-02-20T09:32:59.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:59.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:59.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:59.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:32:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:32:59.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:00.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:32:59.829+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:59.830251+0000) 2024-02-20T09:33:00.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:32:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:32:59.820+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:59.821742+0000) 2024-02-20T09:33:00.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:00.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:00.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:00.224+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:00.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:00.225+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:00.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:00.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:00.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:00.213+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:00.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:00.214+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:00.516+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:00.517530+0000) 2024-02-20T09:33:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:00.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:00.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:00.826 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:00.688+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:00.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:00.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:01.203 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:00.824+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:00.825336+0000) 2024-02-20T09:33:01.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:00.832+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:00.832979+0000) 2024-02-20T09:33:01.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:01.201+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:01.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:01.201+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:01.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:01.201+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:01.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:01.202+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:01.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:01.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:01.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:01.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:01.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:01.212+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:01.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:01.213+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:01.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:01.519+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:01.520122+0000) 2024-02-20T09:33:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:01.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:01.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:01.674+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:01.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:01.674+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:02.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:01.834+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:01.835236+0000) 2024-02-20T09:33:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:01.827+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:01.828117+0000) 2024-02-20T09:33:02.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:02.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:02.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:02.229+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:02.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:02.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:02.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:02.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:02.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:02.521+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.522562+0000) 2024-02-20T09:33:02.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:02.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:02.831 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:02.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:02.832 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:02.689+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:02.832 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:02.689+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:03.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:02.837+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.838818+0000) 2024-02-20T09:33:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:02.829+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.830329+0000) 2024-02-20T09:33:03.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:03.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:03.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:03.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:03.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:03.256+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:03.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:03.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:03.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:03.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:03.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:03.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:03.488 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:03.231+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:03.488 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:03.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:03.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:03.572+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68061 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:03.833 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:03.569+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.569943+0000) 2024-02-20T09:33:03.833 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:03.568+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.568983+0000) 2024-02-20T09:33:03.833 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:03.524+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.525032+0000) 2024-02-20T09:33:03.834 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:03.568+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.569086+0000) 2024-02-20T09:33:03.834 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:03.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:03.834 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:03.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:03.834 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:03.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:03.834 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:03.665+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:04.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:03.838+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.839896+0000) 2024-02-20T09:33:04.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:03.830+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.831493+0000) 2024-02-20T09:33:04.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:04.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:04.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:04.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:04.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:04.233+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:04.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:04.234+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:04.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:04.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:04.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:04.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:04.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:04.209+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:04.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:04.210+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:04.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:04.525+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:04.526138+0000) 2024-02-20T09:33:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:04.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:04.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:04.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:04.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:04.665+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:05.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:04.841+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:04.842548+0000) 2024-02-20T09:33:05.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:04.833+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:04.834649+0000) 2024-02-20T09:33:05.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:05.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:05.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:05.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:05.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:05.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:05.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:05.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:05.518 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:05.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:05.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:05.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:05.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:05.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:05.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:05.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:05.528+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:05.529249+0000) 2024-02-20T09:33:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:05.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:05.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:05.685+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:05.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:05.686+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:06.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:05.844+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:05.845192+0000) 2024-02-20T09:33:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:05.837+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:05.838103+0000) 2024-02-20T09:33:06.533 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:06.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:06.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:06.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:06.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:06.275+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:06.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:06.276+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:06.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:06.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:06.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:06.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:06.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:06.285+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:06.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:06.286+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:06.841 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:06.531+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:06.532644+0000) 2024-02-20T09:33:06.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:06.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:06.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:06.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:06.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:06.696+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:06.842 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:06.697+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:07.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:06.847+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:06.848720+0000) 2024-02-20T09:33:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:06.839+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:06.840727+0000) 2024-02-20T09:33:07.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:07.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:07.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:07.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:07.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:07.296+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:07.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:07.297+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:07.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:07.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:07.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:07.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:07.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:07.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:07.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:07.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:07.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:07.535+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.536110+0000) 2024-02-20T09:33:07.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:07.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:07.845 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:07.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:07.845 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:07.684+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:07.845 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:07.685+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:08.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:07.851+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.852300+0000) 2024-02-20T09:33:08.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:07.842+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.843459+0000) 2024-02-20T09:33:08.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:08.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:08.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:08.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:08.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:08.321+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:08.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:08.322+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:08.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:08.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:08.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:08.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:08.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:08.254+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:08.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:08.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:08.845 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:08.578+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.579012+0000) 2024-02-20T09:33:08.845 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:08.577+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.577986+0000) 2024-02-20T09:33:08.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:08.580+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68102 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:08.846 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:08.538+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.539135+0000) 2024-02-20T09:33:08.846 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:08.577+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.578740+0000) 2024-02-20T09:33:08.846 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:08.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:08.846 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:08.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:08.847 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:08.697+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:08.847 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:08.698+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:09.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:08.852+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.853445+0000) 2024-02-20T09:33:09.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:08.843+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.844433+0000) 2024-02-20T09:33:09.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:09.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:09.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:09.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:09.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:09.355+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:09.494 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:09.356+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:09.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:09.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:09.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:09.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:09.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:09.230+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:09.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:09.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:09.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:09.539+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:09.540175+0000) 2024-02-20T09:33:09.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:09.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:09.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:09.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:09.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:09.664+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:09.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:09.665+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:10.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:09.855+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:09.856846+0000) 2024-02-20T09:33:10.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:09.846+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:09.847841+0000) 2024-02-20T09:33:10.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:10.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:10.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:10.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:10.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:10.393+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:10.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:10.393+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:10.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:10.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:10.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:10.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:10.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:10.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:10.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:10.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:10.852 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:10.542+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:10.543588+0000) 2024-02-20T09:33:10.852 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:10.666+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:10.853 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:10.666+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:10.853 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:10.666+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:10.853 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:10.667+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:11.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:10.850+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:10.851442+0000) 2024-02-20T09:33:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:10.859+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:10.860287+0000) 2024-02-20T09:33:11.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:11.349+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:11.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:11.349+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:11.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:11.349+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:11.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:11.350+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:11.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:11.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:11.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:11.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:11.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:11.279+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:11.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:11.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:11.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:11.546+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:11.546950+0000) 2024-02-20T09:33:11.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:11.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:11.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:11.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:11.855 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:11.657+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:11.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:11.658+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:12.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:11.862+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:11.863621+0000) 2024-02-20T09:33:12.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:11.853+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:11.854161+0000) 2024-02-20T09:33:12.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:12.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:12.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:12.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:12.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:12.380+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:12.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:12.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:12.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:12.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:12.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:12.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:12.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:12.263+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:12.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:12.264+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:12.857 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:12.548+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.549253+0000) 2024-02-20T09:33:12.858 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:12.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:12.858 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:12.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:12.858 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:12.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:12.858 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:12.707+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:13.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:12.866+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.867142+0000) 2024-02-20T09:33:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:12.855+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.856598+0000) 2024-02-20T09:33:13.518 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:13.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:13.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:13.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:13.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:13.379+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:13.519 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:13.380+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:13.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:13.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:13.519 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:13.260+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:13.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:13.261+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:13.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:13.261+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:13.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:13.588+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68141 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:13.859 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:13.585+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.586705+0000) 2024-02-20T09:33:13.859 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:13.585+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.586489+0000) 2024-02-20T09:33:13.859 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:13.551+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.552764+0000) 2024-02-20T09:33:13.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:13.585+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.586265+0000) 2024-02-20T09:33:13.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:13.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:13.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:13.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:13.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:13.750+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:13.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:13.752+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:14.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:13.867+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.868133+0000) 2024-02-20T09:33:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:13.856+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.857662+0000) 2024-02-20T09:33:14.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:14.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:14.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:14.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:14.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:14.361+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:14.555 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:14.362+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:14.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:14.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:14.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:14.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:14.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:14.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:14.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:14.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:14.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:14.552+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:14.553903+0000) 2024-02-20T09:33:14.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:14.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:14.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:14.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:14.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:14.706+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:14.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:14.707+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:15.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:14.869+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:14.870826+0000) 2024-02-20T09:33:15.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:14.858+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:14.859782+0000) 2024-02-20T09:33:15.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:15.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:15.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:15.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:15.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:15.312+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:15.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:15.312+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:15.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:15.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:15.399+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:15.705 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:15.400+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:15.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:15.556+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:15.557208+0000) 2024-02-20T09:33:15.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:15.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:15.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:15.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:15.706 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:15.702+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:15.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:15.862+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:15.863139+0000) 2024-02-20T09:33:15.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:15.703+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:15.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:15.873+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:15.874257+0000) 2024-02-20T09:33:16.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:16.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:16.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:16.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:16.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:16.440+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:16.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:16.440+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:16.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:16.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:16.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:16.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:16.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:16.340+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:16.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:16.341+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:16.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:16.559+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:16.560688+0000) 2024-02-20T09:33:16.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:16.876+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:16.877664+0000) 2024-02-20T09:33:16.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:16.865+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:16.866546+0000) 2024-02-20T09:33:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:16.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:16.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:16.731+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:16.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:16.732+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:17.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:17.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:17.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:17.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:17.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:17.400+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:17.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:17.401+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:17.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:17.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:17.325+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:17.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:17.715 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:17.563+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.564078+0000) 2024-02-20T09:33:17.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:17.880+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.881160+0000) 2024-02-20T09:33:17.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:17.868+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.868968+0000) 2024-02-20T09:33:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:17.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:17.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:17.770+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:17.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:17.771+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:18.711 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:18.597+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68182 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:18.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:18.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:18.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:18.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:18.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:18.430+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:18.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:18.431+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:18.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:18.592+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.593831+0000) 2024-02-20T09:33:18.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:18.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:18.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:18.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:18.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:18.374+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:18.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:18.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:18.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:18.593+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.594564+0000) 2024-02-20T09:33:18.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:18.566+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.567586+0000) 2024-02-20T09:33:18.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:18.593+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.594065+0000) 2024-02-20T09:33:18.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:18.881+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.882779+0000) 2024-02-20T09:33:18.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:18.869+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.870624+0000) 2024-02-20T09:33:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:18.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:18.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:18.789+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:18.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:18.790+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:19.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:19.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:19.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:19.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:19.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:19.477+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:19.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:19.477+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:19.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:19.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:19.403+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:19.404+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:19.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:19.568+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:19.569288+0000) 2024-02-20T09:33:19.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:19.885+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:19.886286+0000) 2024-02-20T09:33:19.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:19.873+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:19.874093+0000) 2024-02-20T09:33:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:19.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:19.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:19.801+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:19.802+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:20.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:20.570+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:20.571737+0000) 2024-02-20T09:33:20.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:20.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:20.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:20.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:20.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:20.513+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:20.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:20.513+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:20.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:20.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:20.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:20.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:20.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:20.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:20.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:20.425+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:20.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:20.888+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:20.889814+0000) 2024-02-20T09:33:20.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:20.876+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:20.877597+0000) 2024-02-20T09:33:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:20.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:20.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:20.838+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:20.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:20.839+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:21.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:21.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:21.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:21.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:21.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:21.552+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:21.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:21.553+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:21.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:21.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:21.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:21.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:21.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:21.444+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:21.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:21.445+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:21.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:21.573+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:21.575293+0000) 2024-02-20T09:33:21.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:21.890+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:21.892029+0000) 2024-02-20T09:33:21.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:21.878+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:21.880165+0000) 2024-02-20T09:33:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:21.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:21.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:21.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:21.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:21.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:21.877+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:22.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:22.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:22.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:22.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:22.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:22.549+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:22.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:22.550+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:22.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:22.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:22.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:22.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:22.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:22.424+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:22.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:22.425+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:22.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:22.576+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.578116+0000) 2024-02-20T09:33:22.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:22.893+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.895174+0000) 2024-02-20T09:33:22.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:22.881+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.883595+0000) 2024-02-20T09:33:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:22.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:22.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:22.903+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:22.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:22.903+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:23.711 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:23.604+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68220 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:23.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:23.579+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.581720+0000) 2024-02-20T09:33:23.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:23.601+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.603754+0000) 2024-02-20T09:33:23.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:23.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:23.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:23.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:23.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:23.565+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:23.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:23.566+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:23.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:23.601+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.603262+0000) 2024-02-20T09:33:23.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:23.474+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:23.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:23.474+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:23.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:23.474+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:23.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:23.475+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:23.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:23.600+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.602667+0000) 2024-02-20T09:33:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:23.894+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.896838+0000) 2024-02-20T09:33:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:23.883+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.885077+0000) 2024-02-20T09:33:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:23.877+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:23.878+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:24.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:24.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:24.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:24.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:24.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:24.484+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:24.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:24.485+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:24.711 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:24.582+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:24.583223+0000) 2024-02-20T09:33:24.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:24.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:24.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:24.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:24.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:24.610+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:24.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:24.611+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:24.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:24.898+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:24.899504+0000) 2024-02-20T09:33:24.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:24.886+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:24.887446+0000) 2024-02-20T09:33:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:24.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:24.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:24.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:24.828+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:24.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:24.829+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:25.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:25.584+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:25.585660+0000) 2024-02-20T09:33:25.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:25.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:25.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:25.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:25.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:25.635+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:25.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:25.636+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:25.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:25.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:25.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:25.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:25.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:25.439+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:25.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:25.440+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:25.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:25.902+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:25.903051+0000) 2024-02-20T09:33:25.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:25.889+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:25.890119+0000) 2024-02-20T09:33:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:25.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:25.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:25.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:25.821+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:25.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:25.822+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:26.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:26.587+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:26.588175+0000) 2024-02-20T09:33:26.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:26.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:26.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:26.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:26.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:26.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:26.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:26.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:26.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:26.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:26.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:26.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:26.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:26.634+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:26.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:26.635+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:26.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:26.904+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:26.905158+0000) 2024-02-20T09:33:26.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:26.891+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:26.892567+0000) 2024-02-20T09:33:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:26.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:26.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:26.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:26.860+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:26.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:26.860+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:27.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:27.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:27.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:27.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:27.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:27.517+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:27.662 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:27.518+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:27.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:27.589+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.590192+0000) 2024-02-20T09:33:27.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:27.894+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.895149+0000) 2024-02-20T09:33:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:27.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:27.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:27.659+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:27.660+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:27.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:27.906+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.907241+0000) 2024-02-20T09:33:27.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:27.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:27.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:27.887+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:27.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:27.888+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:28.712 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:28.613+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68261 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:28.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:28.609+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.610866+0000) 2024-02-20T09:33:28.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:28.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:28.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:28.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:28.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:28.621+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:28.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:28.622+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:28.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:28.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:28.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:28.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:28.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:28.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:28.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:28.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:28.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:28.609+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.610660+0000) 2024-02-20T09:33:28.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:28.592+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.593559+0000) 2024-02-20T09:33:28.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:28.611+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.611964+0000) 2024-02-20T09:33:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:28.907+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.908394+0000) 2024-02-20T09:33:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:28.895+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.896894+0000) 2024-02-20T09:33:28.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:28.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:28.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:28.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:28.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:28.874+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:28.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:28.875+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:29.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:29.594+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:29.595018+0000) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:29.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:29.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:29.613+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:29.614+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:29.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:29.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:29.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:29.473+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:29.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:29.474+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:29.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:29.910+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:29.911012+0000) 2024-02-20T09:33:29.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:29.898+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:29.899190+0000) 2024-02-20T09:33:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:29.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:29.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:29.911+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:29.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:29.912+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:30.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:30.596+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:30.597407+0000) 2024-02-20T09:33:30.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:30.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:30.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:30.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:30.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:30.636+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:30.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:30.637+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:30.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:30.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:30.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:30.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:30.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:30.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:30.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:30.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:30.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:30.913+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:30.914407+0000) 2024-02-20T09:33:30.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:30.901+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:30.901957+0000) 2024-02-20T09:33:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:30.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:30.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:30.899+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:30.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:30.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:31.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:31.599+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:31.600035+0000) 2024-02-20T09:33:31.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:31.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:31.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:31.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:31.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:31.637+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:31.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:31.638+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:31.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:31.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:31.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:31.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:31.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:31.476+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:31.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:31.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:32.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:31.916+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:31.917847+0000) 2024-02-20T09:33:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:31.904+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:31.905341+0000) 2024-02-20T09:33:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:31.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:31.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:31.923+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:32.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:31.924+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:32.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:32.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:32.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:32.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:32.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:32.490+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:32.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:32.491+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:32.644 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:32.601+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.602574+0000) 2024-02-20T09:33:32.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:32.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:32.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:32.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:32.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:32.641+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:32.910 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:32.642+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:33.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:32.919+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.920544+0000) 2024-02-20T09:33:33.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:32.908+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.908968+0000) 2024-02-20T09:33:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:32.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:32.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:32.972+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:33.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:32.973+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:33.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:33.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:33.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:33.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:33.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:33.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:33.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:33.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:33.883 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:33.623+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68299 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:33.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:33.617+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.618859+0000) 2024-02-20T09:33:33.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:33.618+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.618979+0000) 2024-02-20T09:33:33.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:33.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:33.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:33.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:33.885 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:33.652+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:33.885 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:33.653+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:33.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:33.605+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.606248+0000) 2024-02-20T09:33:33.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:33.619+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.619959+0000) 2024-02-20T09:33:34.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:33.920+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.921618+0000) 2024-02-20T09:33:34.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:33.909+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.910768+0000) 2024-02-20T09:33:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.007+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:34.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.008+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:34.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.607+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:34.607951+0000) 2024-02-20T09:33:34.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:34.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:34.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:34.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:34.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:34.445+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:34.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:34.446+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:34.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:34.911+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:34.912792+0000) 2024-02-20T09:33:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:34.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:34.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:34.684+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:34.685+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:34.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:34.922+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:34.923638+0000) 2024-02-20T09:33:35.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:35.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:35.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:35.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:34.980+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:35.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:35.609+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:35.610568+0000) 2024-02-20T09:33:35.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:35.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:35.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:35.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:35.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:35.451+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:35.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:35.452+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:35.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:35.915+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:35.916074+0000) 2024-02-20T09:33:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:35.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:35.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:35.706+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:35.707+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:35.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:35.924+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:35.925806+0000) 2024-02-20T09:33:36.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:36.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:36.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:36.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:36.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:36.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:36.319 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:36.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:36.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:36.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:36.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:36.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:36.615 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:36.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:36.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:36.486+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:36.616 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:36.613+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:36.614159+0000) 2024-02-20T09:33:36.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:36.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:36.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:36.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:36.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:36.733+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:36.920 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:36.733+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:37.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:36.928+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:36.929127+0000) 2024-02-20T09:33:37.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:36.917+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:36.918420+0000) 2024-02-20T09:33:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:37.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:37.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:37.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:37.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:37.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:37.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:37.617+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.618018+0000) 2024-02-20T09:33:37.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:37.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:37.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:37.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:37.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:37.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:37.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:37.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:37.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:37.920+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.921883+0000) 2024-02-20T09:33:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:37.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:37.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:37.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:37.723+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:37.724+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:37.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:37.931+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.932705+0000) 2024-02-20T09:33:38.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:38.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:38.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:38.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:38.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:38.097+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:38.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:38.098+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:38.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:38.631+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68340 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:38.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:38.629+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.630228+0000) 2024-02-20T09:33:38.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:38.620+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.621353+0000) 2024-02-20T09:33:38.653 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:38.629+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.630503+0000) 2024-02-20T09:33:38.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:38.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:38.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:38.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:38.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:38.507+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:38.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:38.507+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:38.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:38.627+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.628186+0000) 2024-02-20T09:33:38.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:38.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:38.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:38.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:38.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:38.752+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:38.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:38.753+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:39.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:38.932+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.933909+0000) 2024-02-20T09:33:39.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:38.922+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.923526+0000) 2024-02-20T09:33:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:39.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:39.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:39.057+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:39.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:39.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:39.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:39.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:39.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:39.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:39.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:39.516+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:39.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:39.517+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:39.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:39.621+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:39.622656+0000) 2024-02-20T09:33:39.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:39.925+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:39.926224+0000) 2024-02-20T09:33:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:39.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:39.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:39.762+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:39.763+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:39.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:39.935+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:39.935972+0000) 2024-02-20T09:33:40.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:40.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:40.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:40.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:40.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:40.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:40.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:40.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:40.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:40.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:40.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:40.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:40.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:40.511+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:40.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:40.512+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:40.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:40.625+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:40.626089+0000) 2024-02-20T09:33:40.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:40.928+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:40.929583+0000) 2024-02-20T09:33:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:40.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:40.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:40.785+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:40.786+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:40.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:40.938+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:40.939338+0000) 2024-02-20T09:33:41.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:41.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:41.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:41.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:41.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:41.130+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:41.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:41.131+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:41.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:41.628+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:41.629613+0000) 2024-02-20T09:33:41.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:41.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:41.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:41.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:41.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:41.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:41.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:41.511+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:41.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:41.932+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:41.932954+0000) 2024-02-20T09:33:41.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:41.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:41.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:41.774+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:41.774+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:41.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:41.941+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:41.942644+0000) 2024-02-20T09:33:42.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:42.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:42.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:42.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:42.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:42.150+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:42.392 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:42.152+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:42.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:42.632+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.633108+0000) 2024-02-20T09:33:42.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:42.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:42.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:42.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:42.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:42.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:42.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:42.487+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:42.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:42.934+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.935105+0000) 2024-02-20T09:33:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:42.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:42.725+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:42.726+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:42.727+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:42.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:42.945+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.946233+0000) 2024-02-20T09:33:43.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:43.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:43.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:43.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:43.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:43.152+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:43.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:43.153+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:43.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:43.640+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68378 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:43.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:43.637+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.637991+0000) 2024-02-20T09:33:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:43.636+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.637114+0000) 2024-02-20T09:33:43.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:43.636+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.637249+0000) 2024-02-20T09:33:43.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:43.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:43.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:43.487+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:43.488+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:43.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:43.636+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.637174+0000) 2024-02-20T09:33:44.204 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:43.935+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.936905+0000) 2024-02-20T09:33:44.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:43.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:44.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:43.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:44.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:43.743+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:44.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:43.744+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:44.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:43.946+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.947876+0000) 2024-02-20T09:33:44.465 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:44.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:44.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:44.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:44.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:44.202+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:44.466 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:44.203+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:44.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:44.638+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:44.639754+0000) 2024-02-20T09:33:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:44.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:44.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:44.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:44.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:44.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:45.216 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:44.938+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:44.939599+0000) 2024-02-20T09:33:45.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:44.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:45.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:44.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:45.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:44.722+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:45.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:44.723+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:45.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:44.950+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:44.951056+0000) 2024-02-20T09:33:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:45.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:45.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:45.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:45.214+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:45.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:45.215+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:45.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:45.642+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:45.643146+0000) 2024-02-20T09:33:45.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:45.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:45.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:45.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:45.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:45.734+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:45.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:45.735+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:45.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:45.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:45.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:45.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:45.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:45.521+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:45.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:45.522+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:45.953+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:45.954237+0000) 2024-02-20T09:33:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:45.942+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:45.943194+0000) 2024-02-20T09:33:46.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:46.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:46.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:46.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:46.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:46.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:46.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:46.234+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:46.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:46.644+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:46.645306+0000) 2024-02-20T09:33:46.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:46.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:46.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:46.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:46.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:46.565+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:46.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:46.566+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:47.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:46.944+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:46.945823+0000) 2024-02-20T09:33:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:46.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:46.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:46.756+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:46.757+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:46.955+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:46.956549+0000) 2024-02-20T09:33:47.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:47.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:47.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:47.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:47.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:47.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:47.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:47.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:47.765 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:47.646+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.647842+0000) 2024-02-20T09:33:47.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:47.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:47.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:47.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:47.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:47.578+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:47.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:47.579+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:47.948+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.949402+0000) 2024-02-20T09:33:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:47.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:47.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:47.763+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:47.764+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:47.957+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.958711+0000) 2024-02-20T09:33:48.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:48.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:48.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:48.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:48.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:48.237+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:48.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:48.238+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:48.772 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:48.649+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68419 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:48.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:48.645+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.646131+0000) 2024-02-20T09:33:48.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:48.645+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.646386+0000) 2024-02-20T09:33:48.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:48.648+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.649198+0000) 2024-02-20T09:33:48.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:48.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:48.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:48.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:48.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:48.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:48.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:48.601+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:48.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:48.644+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.645245+0000) 2024-02-20T09:33:49.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:48.950+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.951251+0000) 2024-02-20T09:33:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:48.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:48.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:48.808+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:49.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:48.809+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:48.959+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.960221+0000) 2024-02-20T09:33:49.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:49.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:49.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:49.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:49.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:49.245+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:49.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:49.245+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:49.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:49.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:49.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:49.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:49.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:49.600+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:49.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:49.601+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:49.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:49.650+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:49.651517+0000) 2024-02-20T09:33:50.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:49.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:49.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:49.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:49.769+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:49.961+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:49.962214+0000) 2024-02-20T09:33:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:49.953+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:49.954830+0000) 2024-02-20T09:33:50.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:50.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:50.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:50.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:50.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:50.254+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:50.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:50.255+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:50.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:50.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:50.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:50.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:50.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:50.582+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:50.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:50.582+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:50.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:50.653+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:50.653971+0000) 2024-02-20T09:33:51.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:50.956+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:50.957457+0000) 2024-02-20T09:33:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:50.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:50.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:50.769+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:50.770+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:51.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:50.963+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:50.964852+0000) 2024-02-20T09:33:51.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:51.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:51.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:51.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:51.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:51.263+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:51.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:51.264+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:51.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:51.655+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:51.656616+0000) 2024-02-20T09:33:51.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:51.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:51.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:51.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:51.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:51.601+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:51.805 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:51.602+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:51.959+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:51.960311+0000) 2024-02-20T09:33:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:51.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:51.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:51.802+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:51.803+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:52.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:51.967+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:51.968437+0000) 2024-02-20T09:33:52.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:52.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:52.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:52.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:52.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:52.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:52.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:52.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:52.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:52.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:52.835 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:52.562+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:52.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:52.564+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:52.836 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:52.658+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:52.659168+0000) 2024-02-20T09:33:53.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:52.962+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:52.963010+0000) 2024-02-20T09:33:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:52.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:52.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:52.833+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:52.834+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:52.971+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:52.972020+0000) 2024-02-20T09:33:53.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:53.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:53.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:53.289+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:53.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:53.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:53.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:53.290+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:53.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:53.657+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68457 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:53.874 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:53.654+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:53.655120+0000) 2024-02-20T09:33:53.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:53.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:53.874 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:53.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:53.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:53.552+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:53.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:53.553+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:53.875 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:53.653+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:53.654510+0000) 2024-02-20T09:33:53.875 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:53.653+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:53.654499+0000) 2024-02-20T09:33:53.875 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:53.660+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:53.660988+0000) 2024-02-20T09:33:54.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:53.963+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:53.964064+0000) 2024-02-20T09:33:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:53.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:53.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:53.871+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:53.872+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:53.972+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:53.973462+0000) 2024-02-20T09:33:54.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:54.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:54.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:54.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:54.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:54.534+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:54.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:54.535+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:54.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:54.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:54.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:54.330+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:54.667 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:54.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:54.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:54.663+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:54.664494+0000) 2024-02-20T09:33:54.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:54.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:54.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:54.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:54.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:54.912+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:54.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:54.913+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:54.976+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:54.976952+0000) 2024-02-20T09:33:55.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:54.965+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:54.966232+0000) 2024-02-20T09:33:55.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:55.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:55.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:55.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:55.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:55.551+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:55.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:55.552+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:55.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:55.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:55.669 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:55.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:55.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:55.302+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:55.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:55.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:55.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:55.666+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:55.667103+0000) 2024-02-20T09:33:55.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:55.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:55.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:55.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:55.918 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:55.915+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:55.919 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:55.916+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:55.979+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:55.980268+0000) 2024-02-20T09:33:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:55.968+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:55.969536+0000) 2024-02-20T09:33:56.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:56.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:56.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:56.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:56.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:56.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:56.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:56.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:56.882 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:56.669+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:56.670500+0000) 2024-02-20T09:33:56.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:56.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:56.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:56.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:56.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:56.566+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:56.883 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:56.567+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:57.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:56.970+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:56.971576+0000) 2024-02-20T09:33:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:56.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:56.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:56.880+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:56.881+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:56.982+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:56.983049+0000) 2024-02-20T09:33:57.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:57.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:57.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:57.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:57.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:57.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:57.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:57.531+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:57.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:57.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:57.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:57.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:57.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:57.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:57.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:57.332+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:57.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:57.671+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:57.672550+0000) 2024-02-20T09:33:57.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:57.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:57.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:57.899+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:57.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:57.900+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:58.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:57.984+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:57.985276+0000) 2024-02-20T09:33:58.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:57.974+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:57.974969+0000) 2024-02-20T09:33:58.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:58.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:58.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:58.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:58.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:58.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:58.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:58.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:58.661 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:58.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:58.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:58.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:58.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:58.339+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:58.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:58.339+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:58.662+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:58.663646+0000) 2024-02-20T09:33:58.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:33:58.665+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68499 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:33:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:58.662+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:58.663536+0000) 2024-02-20T09:33:58.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:58.673+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:58.674140+0000) 2024-02-20T09:33:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:58.662+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:58.663047+0000) 2024-02-20T09:33:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:58.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:58.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:58.905+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:58.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:58.906+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:59.378 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:58.985+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:58.986414+0000) 2024-02-20T09:33:59.378 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:58.975+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:58.976559+0000) 2024-02-20T09:33:59.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:59.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:59.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:59.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:59.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:59.531+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:33:59.679 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:59.532+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:59.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:59.375+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:59.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:59.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:59.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:59.376+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:33:59.680 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:59.376+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:33:59.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:33:59.676+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:59.677751+0000) 2024-02-20T09:33:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:59.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:59.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:59.896+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:33:59.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:59.897+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:00.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:33:59.988+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:59.988953+0000) 2024-02-20T09:34:00.356 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:33:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:33:59.977+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:59.978884+0000) 2024-02-20T09:34:00.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:00.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:00.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:00.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:00.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:00.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:00.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:00.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:00.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:00.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:00.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:00.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:00.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:00.353+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:00.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:00.354+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:00.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:00.679+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:00.680594+0000) 2024-02-20T09:34:01.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:00.981+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:00.982503+0000) 2024-02-20T09:34:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:00.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:00.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:00.940+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:00.941+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:00.990+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:00.991066+0000) 2024-02-20T09:34:01.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:01.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:01.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:01.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:01.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:01.460+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:01.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:01.461+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:01.685 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:01.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:01.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:01.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:01.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:01.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:01.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:01.311+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:01.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:01.682+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:01.683860+0000) 2024-02-20T09:34:02.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:01.984+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:01.985900+0000) 2024-02-20T09:34:02.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:01.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:02.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:01.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:02.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:01.976+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:02.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:01.977+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:02.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:01.992+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:01.993856+0000) 2024-02-20T09:34:02.555 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:02.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:02.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:02.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:02.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:02.502+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:02.556 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:02.503+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:02.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:02.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:02.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:02.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:02.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:02.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:02.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:02.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:02.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:02.686+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:02.687404+0000) 2024-02-20T09:34:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:02.987+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:02.988511+0000) 2024-02-20T09:34:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:02.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:02.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:02.928+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:03.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:02.929+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:03.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:02.995+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:02.995970+0000) 2024-02-20T09:34:03.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:03.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:03.669 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:03.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:03.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:03.485+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:03.670 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:03.486+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:03.670 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:03.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:03.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:03.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:03.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:03.328+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:03.671 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:03.329+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:03.941 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:03.674+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68537 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:03.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:03.669+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:03.670775+0000) 2024-02-20T09:34:03.942 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:03.671+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:03.672771+0000) 2024-02-20T09:34:03.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:03.670+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:03.671412+0000) 2024-02-20T09:34:03.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:03.688+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:03.689267+0000) 2024-02-20T09:34:04.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:03.989+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:03.990027+0000) 2024-02-20T09:34:04.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:03.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:04.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:03.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:04.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:03.939+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:04.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:03.940+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:04.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:03.996+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:03.997270+0000) 2024-02-20T09:34:04.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:04.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:04.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:04.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:04.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:04.510+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:04.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:04.511+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:04.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:04.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:04.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:04.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:04.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:04.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:04.693 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:04.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:04.690+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:04.691331+0000) 2024-02-20T09:34:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:04.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:04.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:04.911+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:04.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:04.911+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:05.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:04.998+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:04.999191+0000) 2024-02-20T09:34:05.282 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:04.991+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:04.992136+0000) 2024-02-20T09:34:05.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:05.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:05.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:05.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:05.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:05.279+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:05.559 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:05.280+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:05.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:05.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:05.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:05.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:05.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:05.520+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:05.560 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:05.521+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:05.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:05.693+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:05.694899+0000) 2024-02-20T09:34:06.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:05.993+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:05.994216+0000) 2024-02-20T09:34:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:05.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:05.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:05.958+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:05.958+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:06.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:06.000+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:06.001712+0000) 2024-02-20T09:34:06.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:06.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:06.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:06.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:06.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:06.265+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:06.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:06.266+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:06.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:06.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:06.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:06.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:06.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:06.500+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:06.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:06.501+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:06.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:06.696+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:06.697570+0000) 2024-02-20T09:34:07.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:06.995+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:06.996811+0000) 2024-02-20T09:34:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:06.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:06.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:06.916+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:06.917+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:07.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:07.004+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:07.005090+0000) 2024-02-20T09:34:07.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:07.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:07.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:07.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:07.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:07.503+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:07.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:07.504+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:07.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:07.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:07.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:07.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:07.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:07.310+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:07.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:07.310+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:07.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:07.700+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:07.700969+0000) 2024-02-20T09:34:08.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:07.998+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:07.999453+0000) 2024-02-20T09:34:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:07.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:07.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:07.956+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:07.957+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:08.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:08.007+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:08.008468+0000) 2024-02-20T09:34:08.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:08.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:08.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:08.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:08.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:08.505+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:08.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:08.506+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:08.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:08.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:08.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:08.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:08.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:08.301+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:08.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:08.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:08.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:08.682+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68578 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:08.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:08.679+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:08.680613+0000) 2024-02-20T09:34:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:08.679+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:08.680426+0000) 2024-02-20T09:34:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:08.679+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:08.680165+0000) 2024-02-20T09:34:08.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:08.701+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:08.702120+0000) 2024-02-20T09:34:09.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:08.999+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:09.000746+0000) 2024-02-20T09:34:09.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:09.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:09.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:09.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:09.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:09.003+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:09.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:09.005+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:09.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:09.009+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:09.010245+0000) 2024-02-20T09:34:09.562 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:09.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:09.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:09.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:09.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:09.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:09.563 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:09.483+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:09.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:09.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:09.563 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:09.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:09.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:09.292+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:09.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:09.293+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:09.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:09.703+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:09.704480+0000) 2024-02-20T09:34:10.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:10.002+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:10.003261+0000) 2024-02-20T09:34:10.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.011+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:10.012395+0000) 2024-02-20T09:34:10.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:10.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:10.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:10.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:10.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:10.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:10.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:10.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:10.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:10.470+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:10.566 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:10.471+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:10.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:10.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:10.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:10.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:10.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:10.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:10.566 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:10.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:10.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:10.705+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:10.706903+0000) 2024-02-20T09:34:11.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:11.005+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:11.006676+0000) 2024-02-20T09:34:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.969+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:10.970+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:11.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:11.015+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:11.015952+0000) 2024-02-20T09:34:11.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:11.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:11.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:11.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:11.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:11.300+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:11.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:11.301+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:11.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:11.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:11.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:11.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:11.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:11.472+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:11.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:11.473+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:11.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:11.708+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:11.709338+0000) 2024-02-20T09:34:12.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:12.008+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:12.009153+0000) 2024-02-20T09:34:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:11.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:11.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:11.941+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:11.941+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:12.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:12.017+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:12.018824+0000) 2024-02-20T09:34:12.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:12.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:12.567 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:12.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:12.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:12.304+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:12.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:12.305+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:12.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:12.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:12.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:12.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:12.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:12.466+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:12.568 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:12.467+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:12.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:12.710+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:12.711875+0000) 2024-02-20T09:34:13.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:13.011+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:13.012775+0000) 2024-02-20T09:34:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:12.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:12.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:12.932+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:12.933+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:13.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:13.020+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:13.021440+0000) 2024-02-20T09:34:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:13.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:13.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:13.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:13.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:13.481+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:13.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:13.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:13.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:13.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:13.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:13.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:13.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:13.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:13.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:13.267+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:13.810 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:13.690+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68616 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:13.811 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:13.687+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:13.688369+0000) 2024-02-20T09:34:13.811 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:13.687+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:13.687955+0000) 2024-02-20T09:34:13.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:13.686+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:13.687779+0000) 2024-02-20T09:34:13.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:13.712+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:13.713150+0000) 2024-02-20T09:34:14.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:14.013+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:14.014315+0000) 2024-02-20T09:34:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:13.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:13.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:13.952+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:13.952+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:14.021+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:14.022845+0000) 2024-02-20T09:34:14.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:14.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:14.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:14.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:14.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:14.223+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:14.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:14.224+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:14.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:14.714+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:14.716308+0000) 2024-02-20T09:34:14.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:14.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:14.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:14.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:14.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:14.496+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:14.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:14.497+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:15.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:15.014+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:15.016782+0000) 2024-02-20T09:34:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:15.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:15.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:15.186+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:15.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:15.187+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:15.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:14.949+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:14.949+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:14.949+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:14.950+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:15.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:15.023+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:15.024953+0000) 2024-02-20T09:34:15.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:15.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:15.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:15.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:15.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:15.455+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:15.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:15.456+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:15.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:15.716+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:15.718541+0000) 2024-02-20T09:34:16.255 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:16.017+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:16.019015+0000) 2024-02-20T09:34:16.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:15.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:16.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:15.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:16.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:15.978+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:16.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:15.979+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:16.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:16.025+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:16.027657+0000) 2024-02-20T09:34:16.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:16.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:16.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:16.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:16.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:16.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:16.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:16.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:16.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:16.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:16.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:16.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:16.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:16.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:16.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:16.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:16.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:16.720+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:16.722023+0000) 2024-02-20T09:34:17.226 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:16.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:17.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:16.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:17.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:16.972+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:17.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:16.974+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:17.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:17.029+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:17.031025+0000) 2024-02-20T09:34:17.227 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:17.020+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:17.021204+0000) 2024-02-20T09:34:17.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:17.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:17.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:17.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:17.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:17.222+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:17.223+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:17.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:17.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:17.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:17.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:17.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:17.376+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:17.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:17.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:17.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:17.724+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:17.725508+0000) 2024-02-20T09:34:18.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:17.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:18.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:17.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:18.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:17.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:18.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:17.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:18.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:18.032+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:18.033424+0000) 2024-02-20T09:34:18.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:18.022+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:18.023242+0000) 2024-02-20T09:34:18.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:18.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:18.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:18.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:18.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:18.345+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:18.525 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:18.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:18.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:18.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:18.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:18.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:18.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:18.262+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:18.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:18.263+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:18.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:18.694+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:18.695851+0000) 2024-02-20T09:34:18.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:18.699+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68657 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:18.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:18.695+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:18.696317+0000) 2024-02-20T09:34:18.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:18.696+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:18.696979+0000) 2024-02-20T09:34:18.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:18.725+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:18.726786+0000) 2024-02-20T09:34:19.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:19.023+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:19.024571+0000) 2024-02-20T09:34:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:19.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:19.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:19.010+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:19.011+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:19.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:19.033+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:19.034732+0000) 2024-02-20T09:34:19.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:19.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:19.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:19.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:19.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:19.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:19.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:19.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:19.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:19.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:19.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:19.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:19.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:19.235+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:19.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:19.236+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:19.728+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:19.729168+0000) 2024-02-20T09:34:20.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:20.036+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:20.037267+0000) 2024-02-20T09:34:20.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:20.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:20.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:20.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:20.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:20.059+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:20.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:20.060+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:20.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:20.026+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:20.027038+0000) 2024-02-20T09:34:20.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:20.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:20.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:20.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:20.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:20.189+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:20.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:20.190+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:20.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:20.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:20.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:20.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:20.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:20.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:20.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:20.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:20.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:20.731+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:20.732755+0000) 2024-02-20T09:34:21.369 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:21.029+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:21.030246+0000) 2024-02-20T09:34:21.369 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:21.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:21.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:21.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:21.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:21.367+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:21.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:21.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:21.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:21.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:21.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:21.191+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:21.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:21.192+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:21.038+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:21.039566+0000) 2024-02-20T09:34:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:21.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:21.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:21.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:21.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:21.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:21.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:21.368+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:21.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:21.733+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:21.734922+0000) 2024-02-20T09:34:22.363 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:22.031+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:22.032822+0000) 2024-02-20T09:34:22.363 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:22.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:22.363 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:22.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:22.363 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:22.361+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:22.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:22.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:22.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:22.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:22.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:22.218+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:22.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:22.219+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:22.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:22.042+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:22.043184+0000) 2024-02-20T09:34:22.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:22.088+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:22.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:22.088+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:22.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:22.088+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:22.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:22.089+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:22.362+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:22.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:22.736+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:22.737740+0000) 2024-02-20T09:34:23.371 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:23.034+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:23.035387+0000) 2024-02-20T09:34:23.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:23.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:23.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:23.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:23.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:23.228+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:23.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:23.229+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:23.044+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:23.045725+0000) 2024-02-20T09:34:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:23.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:23.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:23.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:23.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:23.104+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:23.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:23.105+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:23.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:23.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:23.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:23.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:23.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:23.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:23.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:23.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:23.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:23.703+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:23.704004+0000) 2024-02-20T09:34:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:23.704+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:23.705178+0000) 2024-02-20T09:34:23.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:23.707+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68695 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:23.704+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:23.705378+0000) 2024-02-20T09:34:23.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:23.738+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:23.738960+0000) 2024-02-20T09:34:24.331 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:24.046+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:24.046956+0000) 2024-02-20T09:34:24.331 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:24.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:24.331 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:24.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:24.331 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:24.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:24.331 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:24.075+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:24.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:24.035+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:24.036751+0000) 2024-02-20T09:34:24.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:24.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:24.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:24.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:24.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:24.193+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:24.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:24.193+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:24.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:24.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:24.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:24.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:24.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:24.328+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:24.718 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:24.329+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:24.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:24.740+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:24.741207+0000) 2024-02-20T09:34:25.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:25.049+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:25.050389+0000) 2024-02-20T09:34:25.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:25.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:25.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:25.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:25.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:25.320 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:25.038+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:25.039330+0000) 2024-02-20T09:34:25.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:25.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:25.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:25.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:25.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:25.230+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:25.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:25.231+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:25.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:25.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:25.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:25.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:25.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:25.316+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:25.581 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:25.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:25.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:25.743+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:25.744677+0000) 2024-02-20T09:34:26.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:26.053+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:26.053973+0000) 2024-02-20T09:34:26.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:26.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:26.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:26.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:26.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:26.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:26.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:26.129+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:26.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:26.040+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:26.041791+0000) 2024-02-20T09:34:26.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:26.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:26.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:26.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:26.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:26.270+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:26.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:26.271+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:26.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:26.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:26.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:26.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:26.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:26.278+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:26.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:26.279+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:26.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:26.746+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:26.747068+0000) 2024-02-20T09:34:27.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:27.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:27.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:27.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:27.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:27.268+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:27.396 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:27.269+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:27.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:27.055+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:27.056336+0000) 2024-02-20T09:34:27.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:27.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:27.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:27.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:27.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:27.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:27.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:27.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:27.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:27.043+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:27.044706+0000) 2024-02-20T09:34:27.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:27.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:27.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:27.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:27.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:27.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:27.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:27.248+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:27.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:27.748+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:27.749673+0000) 2024-02-20T09:34:28.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:28.058+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:28.059887+0000) 2024-02-20T09:34:28.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:28.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:28.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:28.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:28.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:28.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:28.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:28.107+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:28.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:28.047+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:28.048206+0000) 2024-02-20T09:34:28.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:28.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:28.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:28.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:28.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:28.268+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:28.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:28.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:28.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:28.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:28.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:28.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:28.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:28.290+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:28.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:28.291+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:28.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:28.712+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:28.713735+0000) 2024-02-20T09:34:28.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:28.712+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:28.713453+0000) 2024-02-20T09:34:28.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:28.716+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68736 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:28.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:28.712+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:28.713061+0000) 2024-02-20T09:34:28.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:28.750+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:28.750951+0000) 2024-02-20T09:34:29.343 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:29.060+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:29.061042+0000) 2024-02-20T09:34:29.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:29.095+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:29.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:29.095+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:29.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:29.095+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:29.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:29.096+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:29.344 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:29.048+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:29.049523+0000) 2024-02-20T09:34:29.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:29.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:29.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:29.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:29.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:29.242+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:29.345 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:29.243+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:29.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:29.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:29.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:29.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:29.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:29.341+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:29.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:29.342+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:29.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:29.752+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:29.753571+0000) 2024-02-20T09:34:30.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:30.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:30.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:30.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:30.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:30.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:30.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:30.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:30.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:30.063+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:30.064573+0000) 2024-02-20T09:34:30.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:30.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:30.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:30.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:30.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:30.297+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:30.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:30.297+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:30.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:30.050+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:30.051918+0000) 2024-02-20T09:34:30.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:30.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:30.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:30.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:30.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:30.220+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:30.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:30.221+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:30.651 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-20T09:34:30.666 INFO:tasks.cephadm:Teardown begin 2024-02-20T09:34:30.666 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-20T09:34:30.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-20T09:34:30.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:34:30.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:30.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:30.755+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:30.756061+0000) 2024-02-20T09:34:31.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:31.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:31.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:31.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:31.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:31.002+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:31.326 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:31.003+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:31.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:31.066+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:31.066970+0000) 2024-02-20T09:34:31.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:31.054+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:31.055548+0000) 2024-02-20T09:34:31.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:31.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:31.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:31.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:31.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:31.247+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:31.327 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:31.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:31.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:31.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:31.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:31.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:31.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:31.324+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:31.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:31.325+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:31.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:31.757+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:31.758479+0000) 2024-02-20T09:34:32.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:32.057+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:32.058198+0000) 2024-02-20T09:34:32.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:32.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:32.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:32.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:32.258 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:32.255+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:23:59.918372+0000 (oldest deadline 2024-02-20T09:24:19.918372+0000) 2024-02-20T09:34:32.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:32.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:32.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:32.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:32.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:32.035+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:32.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:32.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:32.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:32.068+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:32.069621+0000) 2024-02-20T09:34:32.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:32.256+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:32.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:32.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:32.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:32.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:32.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:32.348+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:32.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:32.349+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:32.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:32.759+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:32.760921+0000) 2024-02-20T09:34:33.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:33.060+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:33.061575+0000) 2024-02-20T09:34:33.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:33.271+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:33.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:33.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:33.038+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:33.039+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:33.306 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:33.070+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:33.071832+0000) 2024-02-20T09:34:33.589 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:33.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:33.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:33.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:33.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:33.303+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:33.590 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:33.304+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:33.792 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:34:33.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:33.725+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68775 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:33.848 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:33.721+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:33.722086+0000) 2024-02-20T09:34:33.849 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:33.720+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:33.721526+0000) 2024-02-20T09:34:33.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:33.722+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:33.723106+0000) 2024-02-20T09:34:33.849 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:33.761+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:33.762476+0000) 2024-02-20T09:34:34.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:34.061+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:34.062817+0000) 2024-02-20T09:34:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:34.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:34.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:34.069+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:34.070+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:34.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:34.072+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:34.072973+0000) 2024-02-20T09:34:34.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:34.317+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:34.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:34.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:34.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:34.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:34.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:34.257+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:34.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:34.258+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:34.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:34.765+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:34.766032+0000) 2024-02-20T09:34:35.396 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:35.064+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:35.064974+0000) 2024-02-20T09:34:35.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:35.305+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:35.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:35.074+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:35.075112+0000) 2024-02-20T09:34:35.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:35.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:35.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:35.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:35.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:35.102+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:35.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:35.103+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:35.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:35.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:35.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:35.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:35.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:35.224+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:35.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:35.225+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:36.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:35.768+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:35.769620+0000) 2024-02-20T09:34:36.396 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:36.066+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:36.067637+0000) 2024-02-20T09:34:36.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:36.302+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:36.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:36.076+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:36.077815+0000) 2024-02-20T09:34:36.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:36.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:36.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:36.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:36.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:36.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:36.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:36.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:36.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:36.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:36.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:36.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:36.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:36.187+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:36.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:36.188+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:37.071 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:36.772+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:36.773214+0000) 2024-02-20T09:34:37.332 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:37.069+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:37.069969+0000) 2024-02-20T09:34:37.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:37.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:37.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:37.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:37.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:37.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:37.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:37.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:37.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:37.080+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:37.081323+0000) 2024-02-20T09:34:37.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:37.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:37.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:37.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:37.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:37.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:37.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:37.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:37.594 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:37.330+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:37.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:37.774+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:37.775439+0000) 2024-02-20T09:34:38.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:38.071+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:38.072566+0000) 2024-02-20T09:34:38.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:38.318+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:38.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:38.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:38.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:38.134+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:38.134+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:38.082+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:38.083465+0000) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:38.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:38.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:38.159+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:38.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:38.160+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:39.075 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:38.733+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68816 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:39.075 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:38.730+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:38.731094+0000) 2024-02-20T09:34:39.076 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:38.730+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:38.731452+0000) 2024-02-20T09:34:39.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:38.730+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:38.731678+0000) 2024-02-20T09:34:39.076 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:38.775+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:38.776831+0000) 2024-02-20T09:34:39.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:39.073+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:39.073968+0000) 2024-02-20T09:34:39.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:39.313+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:39.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:39.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:39.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:39.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:39.099+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:39.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:39.084+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:39.085413+0000) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:39.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:39.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:39.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:39.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:39.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:40.077 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:39.778+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:39.779328+0000) 2024-02-20T09:34:40.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:40.075+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:40.076713+0000) 2024-02-20T09:34:40.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:40.321+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:40.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:40.088+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:40.089007+0000) 2024-02-20T09:34:40.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:40.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:40.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:40.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:40.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:40.224+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:40.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:40.225+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:40.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:40.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:40.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:40.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:40.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:40.128+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:40.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:40.129+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:41.080 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:40.780+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:40.781842+0000) 2024-02-20T09:34:41.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:41.078+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:41.079272+0000) 2024-02-20T09:34:41.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:41.306+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:41.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:41.090+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:41.091660+0000) 2024-02-20T09:34:41.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:41.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:41.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:41.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:41.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:41.257+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.364544+0000 (oldest deadline 2024-02-20T09:24:29.364544+0000) 2024-02-20T09:34:41.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:41.257+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:41.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:41.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:41.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:41.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:41.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:41.079+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:41.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:41.080+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:42.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:41.784+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:41.785467+0000) 2024-02-20T09:34:42.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:42.093+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:42.094207+0000) 2024-02-20T09:34:42.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:42.275+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:42.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:42.081+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:42.082279+0000) 2024-02-20T09:34:42.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:42.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:42.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:42.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:42.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:42.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:42.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:42.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:42.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:42.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:42.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:34:42.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:43.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:42.787+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:42.788114+0000) 2024-02-20T09:34:43.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:43.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:43.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:43.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:43.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:43.095+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:43.096245+0000) 2024-02-20T09:34:43.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:43.289+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:43.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:43.084+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:43.085078+0000) 2024-02-20T09:34:43.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:43.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:43.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:43.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:43.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:43.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:44.084 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:43.739+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:43.740233+0000) 2024-02-20T09:34:44.085 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:43.738+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:43.739902+0000) 2024-02-20T09:34:44.085 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:43.742+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68854 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:44.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:43.738+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:43.739406+0000) 2024-02-20T09:34:44.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:43.788+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:43.789410+0000) 2024-02-20T09:34:44.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:44.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:44.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:44.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:44.086 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:44.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:24:09.770256+0000 (oldest deadline 2024-02-20T09:24:29.770256+0000) 2024-02-20T09:34:44.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:44.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:44.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:44.096+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:44.097676+0000) 2024-02-20T09:34:44.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:44.327+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:44.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:44.085+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:44.086565+0000) 2024-02-20T09:34:44.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:44.245+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:45.060 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:44.790+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:44.791354+0000) 2024-02-20T09:34:45.397 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:45.058+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:45.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:45.099+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:45.100686+0000) 2024-02-20T09:34:45.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:45.296+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:45.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:45.089+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:45.090102+0000) 2024-02-20T09:34:45.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:45.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:45.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:34:46.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:45.793+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:45.794034+0000) 2024-02-20T09:34:46.095 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:46.024+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:46.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:46.103+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:46.104114+0000) 2024-02-20T09:34:46.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:46.340+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:46.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:46.092+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:46.093637+0000) 2024-02-20T09:34:46.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:46.232+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:47.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:46.796+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:46.797711+0000) 2024-02-20T09:34:47.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:47.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:47.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:47.106+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:47.107649+0000) 2024-02-20T09:34:47.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:47.293+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:47.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:47.096+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:47.097123+0000) 2024-02-20T09:34:47.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:47.239+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:48.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:47.799+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:47.800259+0000) 2024-02-20T09:34:48.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:48.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:48.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:48.109+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:48.110209+0000) 2024-02-20T09:34:48.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:48.303+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:48.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:48.098+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:48.099766+0000) 2024-02-20T09:34:48.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:48.227+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:49.102 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:48.751+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68895 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:49.102 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:48.747+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:48.748742+0000) 2024-02-20T09:34:49.102 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:48.746+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:48.747770+0000) 2024-02-20T09:34:49.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:48.801+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:48.802312+0000) 2024-02-20T09:34:49.103 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:48.746+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:48.747429+0000) 2024-02-20T09:34:49.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:49.110+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:49.111575+0000) 2024-02-20T09:34:49.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:49.305+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:49.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:49.100+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:49.100971+0000) 2024-02-20T09:34:49.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:49.206+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:49.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:49.116+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:50.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:49.803+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:49.804714+0000) 2024-02-20T09:34:50.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:50.112+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:50.113827+0000) 2024-02-20T09:34:50.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:50.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:50.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:50.102+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:50.103585+0000) 2024-02-20T09:34:50.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:50.236+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:50.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:50.114+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:51.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:50.807+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:50.808354+0000) 2024-02-20T09:34:51.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:51.115+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:51.116827+0000) 2024-02-20T09:34:51.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:51.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:51.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:51.105+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:51.106167+0000) 2024-02-20T09:34:51.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:51.250+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:51.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:51.121+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:52.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:51.809+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:51.810808+0000) 2024-02-20T09:34:52.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:52.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:52.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:52.119+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:52.120372+0000) 2024-02-20T09:34:52.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:52.215+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:52.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:52.107+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:52.108761+0000) 2024-02-20T09:34:52.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:52.271+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:53.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:52.813+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:52.814178+0000) 2024-02-20T09:34:53.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:53.086+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:53.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:53.123+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:53.123961+0000) 2024-02-20T09:34:53.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:53.206+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:53.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:53.111+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:53.112135+0000) 2024-02-20T09:34:53.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:53.280+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:54.065 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:53.756+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:53.757300+0000) 2024-02-20T09:34:54.066 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:53.758+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:53.759487+0000) 2024-02-20T09:34:54.066 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:53.760+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68933 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:54.066 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:53.755+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:53.756493+0000) 2024-02-20T09:34:54.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:53.815+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:53.816213+0000) 2024-02-20T09:34:54.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:54.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:54.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:54.124+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:54.125142+0000) 2024-02-20T09:34:54.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:54.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:54.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:54.112+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:54.113483+0000) 2024-02-20T09:34:54.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:54.255+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:55.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:54.817+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:54.818703+0000) 2024-02-20T09:34:55.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:55.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:55.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:55.126+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:55.127652+0000) 2024-02-20T09:34:55.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:55.191+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:55.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:55.114+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:55.115924+0000) 2024-02-20T09:34:55.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:55.231+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:56.119 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:55.820+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:55.821167+0000) 2024-02-20T09:34:56.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:56.139+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:56.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:56.117+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:56.118372+0000) 2024-02-20T09:34:56.399 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:56.275+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:56.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:56.129+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:56.130099+0000) 2024-02-20T09:34:56.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:56.168+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:57.106 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:56.823+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:56.824670+0000) 2024-02-20T09:34:57.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:57.104+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:57.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:57.132+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:57.133648+0000) 2024-02-20T09:34:57.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:57.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:57.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:57.120+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:57.121116+0000) 2024-02-20T09:34:57.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:57.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:57.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:57.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:57.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:57.237+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:57.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:57.237+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:57.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:34:57.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:58.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:57.826+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:57.827178+0000) 2024-02-20T09:34:58.100 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:58.097+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:58.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:58.135+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:58.135970+0000) 2024-02-20T09:34:58.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:58.227+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:58.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:58.122+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:58.123775+0000) 2024-02-20T09:34:58.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:58.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:58.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:58.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:58.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:58.241+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:58.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:58.242+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:59.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:34:58.769+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 68974 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:34:59.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:58.764+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:58.765794+0000) 2024-02-20T09:34:59.127 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:58.764+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:58.765773+0000) 2024-02-20T09:34:59.127 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:58.765+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:58.766805+0000) 2024-02-20T09:34:59.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:58.828+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:58.828953+0000) 2024-02-20T09:34:59.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:59.136+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:59.137216+0000) 2024-02-20T09:34:59.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:34:59.195+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:59.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:59.123+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:34:59.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:59.124+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:59.125030+0000) 2024-02-20T09:34:59.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:59.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:59.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:59.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:59.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:59.287+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:34:59.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:34:59.288+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:00.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:34:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:34:59.830+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:34:59.831407+0000) 2024-02-20T09:35:00.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:00.127+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:00.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:00.138+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:00.139432+0000) 2024-02-20T09:35:00.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:00.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:00.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:00.126+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:00.127332+0000) 2024-02-20T09:35:00.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:00.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:00.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:00.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:00.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:00.313+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:00.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:00.314+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:00.928 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:01.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:00.833+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:00.834758+0000) 2024-02-20T09:35:01.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:01.109+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:01.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:01.140+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:01.141567+0000) 2024-02-20T09:35:01.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:01.155+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:01.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:01.128+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:01.129456+0000) 2024-02-20T09:35:01.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:01.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:01.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:01.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:01.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:01.271+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:01.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:01.272+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:02.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:01.837+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:01.838062+0000) 2024-02-20T09:35:02.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:02.132+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:02.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:02.142+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:02.143816+0000) 2024-02-20T09:35:02.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:02.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:02.400 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:02.130+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:02.131654+0000) 2024-02-20T09:35:02.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:02.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:02.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:02.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:02.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:02.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:02.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:02.292+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:03.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:02.840+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:02.841687+0000) 2024-02-20T09:35:03.101 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:03.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:03.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:03.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:03.099+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:03.099+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:03.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:03.145+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:03.145954+0000) 2024-02-20T09:35:03.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:03.133+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:03.134409+0000) 2024-02-20T09:35:03.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:03.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:03.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:03.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:03.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:03.326+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:03.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:03.327+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:03.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:03.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:04.000 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:04.000 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:35:04.001 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-20T09:35:04.001 DEBUG:teuthology.orchestra.run.smithi055:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T09:35:04.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:04.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:04.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:03.777+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69012 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:04.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:03.774+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:03.775292+0000) 2024-02-20T09:35:04.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:03.773+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:03.774677+0000) 2024-02-20T09:35:04.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:03.773+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:03.774363+0000) 2024-02-20T09:35:04.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:03.842+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:03.843423+0000) 2024-02-20T09:35:04.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:04.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:04.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:04.135+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:04.136144+0000) 2024-02-20T09:35:04.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:04.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:04.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:04.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:04.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:04.145+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:04.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:04.146+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:04.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:04.146+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:04.147499+0000) 2024-02-20T09:35:04.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:04.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:04.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:04.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:04.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:04.301+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:04.623 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:04.302+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:04.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:04.845+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:04.845961+0000) 2024-02-20T09:35:05.400 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:05.033+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:05.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:05.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:05.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:05.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:05.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:05.124+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:05.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:05.125+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:05.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:05.149+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:05.150162+0000) 2024-02-20T09:35:05.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:05.137+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:05.138705+0000) 2024-02-20T09:35:05.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:05.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:05.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:05.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:05.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:05.265+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:05.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:05.267+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:06.115 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:05.848+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:05.849553+0000) 2024-02-20T09:35:06.116 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:06.005+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:06.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:06.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:06.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:06.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:06.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:06.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:06.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:06.152+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:06.153726+0000) 2024-02-20T09:35:06.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:06.139+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:06.140870+0000) 2024-02-20T09:35:06.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:06.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:06.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:06.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:06.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:06.269+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:06.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:06.269+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:07.100 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:07.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:06.851+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:06.852053+0000) 2024-02-20T09:35:07.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:06.989+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:07.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:07.137+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:07.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:07.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:07.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:07.138+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:07.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:07.138+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:07.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:07.154+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:07.156739+0000) 2024-02-20T09:35:07.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:07.142+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:07.144072+0000) 2024-02-20T09:35:07.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:07.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:07.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:07.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:07.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:07.246+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:07.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:07.247+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:08.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:07.852+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:07.854478+0000) 2024-02-20T09:35:08.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:07.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:08.139 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:07.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:08.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:07.971+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:08.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:07.971+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:08.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:08.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:08.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:08.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:08.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:08.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:08.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:08.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:08.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:08.157+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:08.159367+0000) 2024-02-20T09:35:08.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:08.144+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:08.146259+0000) 2024-02-20T09:35:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:08.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:08.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:08.210+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:08.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:08.211+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:09.124 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:08.780+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:08.782565+0000) 2024-02-20T09:35:09.125 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:08.781+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:08.783526+0000) 2024-02-20T09:35:09.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:08.784+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69052 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:09.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:08.780+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:08.782853+0000) 2024-02-20T09:35:09.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:08.854+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:08.856074+0000) 2024-02-20T09:35:09.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:08.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:09.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:08.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:09.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:08.926+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:09.126 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:08.928+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:09.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:09.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:09.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:09.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:09.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:09.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:09.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:09.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:09.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:09.158+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:09.160782+0000) 2024-02-20T09:35:09.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:09.145+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:09.147711+0000) 2024-02-20T09:35:09.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:09.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:09.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:09.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:09.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:09.203+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:09.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:09.203+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:10.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:10.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:10.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:10.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:10.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:10.076+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:10.151 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:10.076+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:10.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:09.857+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:09.858467+0000) 2024-02-20T09:35:10.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:09.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:10.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:09.964+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:10.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:09.966+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:10.152 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:09.966+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:10.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:10.162+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:10.163214+0000) 2024-02-20T09:35:10.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:10.147+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:10.149735+0000) 2024-02-20T09:35:10.467 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:10.217+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:10.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:10.217+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:10.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:10.217+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:10.468 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:10.218+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:11.153 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:11.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:11.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:11.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:11.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:11.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:11.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:11.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:11.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:10.860+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:10.861089+0000) 2024-02-20T09:35:11.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:11.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:11.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.003+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:11.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.004+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:11.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:11.164+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:11.165350+0000) 2024-02-20T09:35:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:11.151+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:11.152306+0000) 2024-02-20T09:35:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:11.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:11.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:11.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:11.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:11.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:11.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:12.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:12.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:12.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:12.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:12.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:12.016+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:12.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:12.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:12.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.862+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:11.863825+0000) 2024-02-20T09:35:12.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:12.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:12.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:12.158 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:11.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:12.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:12.167+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:12.168566+0000) 2024-02-20T09:35:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:12.154+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:12.154985+0000) 2024-02-20T09:35:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:12.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:12.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:12.188+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:12.189+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:13.144 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:13.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:13.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:13.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:13.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:13.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:13.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:13.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:13.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:12.865+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:12.866363+0000) 2024-02-20T09:35:13.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:13.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:13.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.017+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:13.146 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.018+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:13.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:13.171+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:13.172094+0000) 2024-02-20T09:35:13.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:13.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:13.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:13.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:13.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:13.142+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:13.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:13.143+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:13.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:13.157+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:13.158110+0000) 2024-02-20T09:35:14.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:13.790+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:13.791084+0000) 2024-02-20T09:35:14.056 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:13.790+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:13.791668+0000) 2024-02-20T09:35:14.056 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:13.794+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69091 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:14.056 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.792+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:13.792950+0000) 2024-02-20T09:35:14.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.867+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:13.868117+0000) 2024-02-20T09:35:14.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:14.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:14.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.997+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:14.057 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:13.998+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:14.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:14.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:14.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:14.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:14.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:14.094+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:14.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:14.095+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:14.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:14.158+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:14.159692+0000) 2024-02-20T09:35:14.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:14.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:14.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:14.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:14.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:14.050+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:14.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:14.051+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:14.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:14.172+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:14.173503+0000) 2024-02-20T09:35:15.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:15.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:15.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:15.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:15.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:15.085+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:15.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:15.086+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:15.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:15.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:15.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:15.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:15.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:15.017+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:15.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:15.017+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:15.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:14.869+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:14.870533+0000) 2024-02-20T09:35:15.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:15.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:15.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:15.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:15.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:15.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:15.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:15.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:15.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:15.175+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:15.176100+0000) 2024-02-20T09:35:15.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:15.161+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:15.162182+0000) 2024-02-20T09:35:16.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:16.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:16.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:16.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:16.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:16.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:16.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:16.008+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:16.166 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:16.009+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:16.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:16.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:16.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:16.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:16.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:16.066+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:16.167 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:16.067+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:16.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:15.871+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:15.872667+0000) 2024-02-20T09:35:16.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:16.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:16.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:16.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:16.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:16.058+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:16.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:16.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:16.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:16.163+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:16.164653+0000) 2024-02-20T09:35:16.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:16.177+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:16.178646+0000) 2024-02-20T09:35:17.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:17.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:17.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:17.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:17.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:17.075+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:17.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:17.076+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:17.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:16.874+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:16.875196+0000) 2024-02-20T09:35:17.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:17.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:17.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:17.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:17.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:17.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:17.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:17.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:17.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:17.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:17.170 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:17.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:17.170 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:17.040+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:17.170 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:17.041+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:17.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:17.179+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:17.180781+0000) 2024-02-20T09:35:17.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:17.166+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:17.166958+0000) 2024-02-20T09:35:18.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:17.877+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:17.877957+0000) 2024-02-20T09:35:18.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:18.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:18.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:18.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:18.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:18.042+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:18.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:18.043+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:18.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:18.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:18.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:18.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:18.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:18.074+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:18.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:18.074+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:18.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:18.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:18.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:18.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:18.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:18.059+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:18.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:18.060+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:18.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:18.183+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:18.184374+0000) 2024-02-20T09:35:18.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:18.168+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:18.169870+0000) 2024-02-20T09:35:19.082 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:18.798+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:18.799397+0000) 2024-02-20T09:35:19.083 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:18.803+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69131 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:19.083 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:18.798+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:18.799806+0000) 2024-02-20T09:35:19.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:18.878+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:18.879283+0000) 2024-02-20T09:35:19.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:19.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:19.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:18.799+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:18.800847+0000) 2024-02-20T09:35:19.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:19.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:19.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:19.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:19.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:19.036+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:19.085 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:19.037+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:19.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:19.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:19.170+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:19.171446+0000) 2024-02-20T09:35:19.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:19.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:19.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:19.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:19.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:19.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:19.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:19.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:19.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:19.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:19.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:19.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:19.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:19.092+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:19.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:19.185+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:19.186013+0000) 2024-02-20T09:35:20.135 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:20.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:20.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:20.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:20.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:20.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:20.136 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:20.022+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:20.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:19.880+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:19.881893+0000) 2024-02-20T09:35:20.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:20.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:20.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:20.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:20.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:20.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:20.137 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:20.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:20.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:20.173+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:20.174276+0000) 2024-02-20T09:35:20.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:20.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:20.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:20.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:20.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:20.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:20.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:20.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:20.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:20.187+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:20.188430+0000) 2024-02-20T09:35:21.029 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:20.883+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:20.884379+0000) 2024-02-20T09:35:21.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:21.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:21.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:21.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:21.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:21.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:21.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:21.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:21.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:21.189+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:21.190874+0000) 2024-02-20T09:35:21.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:21.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:21.404 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.027+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:21.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.028+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:21.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.175+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:21.176702+0000) 2024-02-20T09:35:21.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:21.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:21.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:21.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:21.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:21.067+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:21.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:21.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:22.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:22.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:22.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:22.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:22.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:22.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:22.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:22.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:22.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:22.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:22.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.998+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:22.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:21.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:22.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:21.886+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:21.886951+0000) 2024-02-20T09:35:22.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:22.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:22.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:22.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:22.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:22.104+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:22.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:22.105+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:22.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:22.192+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:22.193164+0000) 2024-02-20T09:35:22.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:22.178+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:22.179182+0000) 2024-02-20T09:35:23.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:23.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:23.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:23.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:23.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:23.121+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:23.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:23.122+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:23.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:22.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:23.184 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:22.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:23.184 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:22.996+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:23.184 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:22.998+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:23.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:22.888+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:22.889418+0000) 2024-02-20T09:35:23.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:23.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:23.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:23.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:23.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:23.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:23.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:23.060+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:23.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:23.195+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:23.196744+0000) 2024-02-20T09:35:23.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:23.180+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:23.181758+0000) 2024-02-20T09:35:24.094 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:23.812+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69170 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:24.094 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:23.807+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:23.808396+0000) 2024-02-20T09:35:24.095 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:23.808+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:23.809269+0000) 2024-02-20T09:35:24.095 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:23.890+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:23.891024+0000) 2024-02-20T09:35:24.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:23.808+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:23.809159+0000) 2024-02-20T09:35:24.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:23.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:24.095 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:23.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:24.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:23.957+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:24.096 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:23.957+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:24.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:24.182+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:24.183483+0000) 2024-02-20T09:35:24.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:24.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:24.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:24.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:24.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:24.092+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:24.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:24.093+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:24.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:24.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:24.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:24.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:24.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:24.113+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:24.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:24.114+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:24.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:24.197+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:24.198048+0000) 2024-02-20T09:35:25.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:24.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:25.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:24.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:25.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:24.920+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:25.166 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:24.921+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:25.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:24.892+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:24.893426+0000) 2024-02-20T09:35:25.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:25.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:25.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:25.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:25.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:25.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:25.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:25.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:25.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:25.185+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:25.185967+0000) 2024-02-20T09:35:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:25.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:25.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:25.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:25.164+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:25.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:25.199+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:25.200673+0000) 2024-02-20T09:35:26.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:25.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:26.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:25.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:26.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:25.884+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:26.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:25.884+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:26.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:25.894+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:25.895554+0000) 2024-02-20T09:35:26.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:26.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:26.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:26.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:26.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:26.068+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:26.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:26.068+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:26.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:26.187+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:26.188073+0000) 2024-02-20T09:35:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:26.203+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:26.204108+0000) 2024-02-20T09:35:26.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:26.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:26.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:26.204+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:26.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:26.205+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:26.896+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:26.897671+0000) 2024-02-20T09:35:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:27.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:27.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:27.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:27.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:27.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:26.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:26.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:26.897+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:27.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:26.898+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:27.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:27.189+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:27.190903+0000) 2024-02-20T09:35:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:27.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:27.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:27.200+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:27.201+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:27.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:27.206+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:27.207834+0000) 2024-02-20T09:35:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:27.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:27.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:28.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:27.896+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:28.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:27.897+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:28.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:27.899+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:27.900365+0000) 2024-02-20T09:35:28.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:28.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:28.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:28.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:28.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:28.136+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:28.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:28.137+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:28.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:28.193+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:28.194544+0000) 2024-02-20T09:35:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:28.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:28.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:28.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:28.190+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:28.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:28.210+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:28.211374+0000) 2024-02-20T09:35:29.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:28.822+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69210 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:29.092 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:28.816+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:28.817702+0000) 2024-02-20T09:35:29.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:28.817+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:28.818280+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:28.901+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:28.902380+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:29.088+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:29.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:29.089+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:28.816+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:28.817579+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:28.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:29.093 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:28.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:29.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:28.859+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:29.094 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:28.860+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:29.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:29.195+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:29.196034+0000) 2024-02-20T09:35:29.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:29.090+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:29.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:29.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:29.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:29.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:29.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:29.173+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:29.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:29.174+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:29.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:29.212+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:29.212959+0000) 2024-02-20T09:35:30.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:29.850+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:30.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:29.850+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:30.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:29.850+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:30.121 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:29.850+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:30.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:29.903+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:29.904596+0000) 2024-02-20T09:35:30.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:30.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:30.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:30.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:30.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:30.118+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:30.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:30.197+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:30.198455+0000) 2024-02-20T09:35:30.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:30.120+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:30.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:30.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:30.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:30.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:30.150+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:30.150+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:30.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:30.214+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:30.215235+0000) 2024-02-20T09:35:30.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:30.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:30.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:30.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:30.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:30.898+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:30.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:30.899+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:31.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:31.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:31.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:31.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:31.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:31.202 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:31.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:31.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:30.906+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:30.907114+0000) 2024-02-20T09:35:31.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:31.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:31.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:31.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:31.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:31.108+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:31.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:31.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:31.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:31.216+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:31.217763+0000) 2024-02-20T09:35:31.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:31.200+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:31.201060+0000) 2024-02-20T09:35:32.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:32.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:32.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:32.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:32.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:32.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:32.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:32.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:32.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:31.944+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:32.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:31.944+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:32.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:31.944+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:32.206 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:31.945+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:32.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:31.908+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:31.909807+0000) 2024-02-20T09:35:32.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:32.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:32.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:32.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:32.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:32.116+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:32.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:32.117+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:32.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:32.219+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:32.220313+0000) 2024-02-20T09:35:32.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:32.202+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:32.203218+0000) 2024-02-20T09:35:33.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:33.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:33.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:33.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:33.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:33.107+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:33.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:33.108+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:33.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:32.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:33.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:32.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:33.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:32.947+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:33.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:32.948+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:33.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:32.911+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:32.912253+0000) 2024-02-20T09:35:33.208 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:33.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:33.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:33.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:33.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:33.146+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:33.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:33.146+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:33.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:33.222+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:33.223064+0000) 2024-02-20T09:35:33.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:33.204+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:33.205774+0000) 2024-02-20T09:35:34.133 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:33.826+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:33.827664+0000) 2024-02-20T09:35:34.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:33.826+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:33.827659+0000) 2024-02-20T09:35:34.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:33.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:34.134 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:33.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:34.134 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:33.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:34.134 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:33.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:34.134 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:33.830+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69253 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:34.134 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:33.827+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:33.828040+0000) 2024-02-20T09:35:34.135 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:33.912+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:33.913705+0000) 2024-02-20T09:35:34.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:34.403 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:34.206+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:34.207729+0000) 2024-02-20T09:35:34.403 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:34.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:34.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:34.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:34.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:34.130+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:34.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:34.131+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:34.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:34.223+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:34.224506+0000) 2024-02-20T09:35:34.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:34.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:34.404 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:34.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:34.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:34.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:34.405 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:34.140+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:35.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:35.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:35.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:35.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:35.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:35.105+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:35.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:35.106+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:35.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:34.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:35.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:34.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:35.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:34.975+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:35.212 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:34.976+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:35.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:34.915+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:34.916306+0000) 2024-02-20T09:35:35.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:35.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:35.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:35.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:35.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:35.148+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:35.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:35.149+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:35.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:35.225+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:35.226450+0000) 2024-02-20T09:35:35.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:35.209+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:35.210116+0000) 2024-02-20T09:35:36.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:36.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:36.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:36.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:36.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:36.142+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:36.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:36.143+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:36.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:36.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:36.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:36.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:35.917+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:35.918848+0000) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:36.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:36.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:36.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:36.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:36.106+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:36.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:36.107+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:36.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:36.227+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:36.228617+0000) 2024-02-20T09:35:36.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:36.211+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:36.212820+0000) 2024-02-20T09:35:37.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:37.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:37.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:37.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:37.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:37.152+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:37.217 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:37.153+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:37.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:37.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:37.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:37.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:37.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:37.030+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:37.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:37.031+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:37.218 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:36.921+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:36.921974+0000) 2024-02-20T09:35:37.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:37.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:37.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:37.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:37.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:37.059+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:37.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:37.059+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:37.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:37.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:37.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:37.308 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:35:37.309 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-20T09:35:37.310 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543/remote/smithi055/crash 2024-02-20T09:35:37.311 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/crash -- . 2024-02-20T09:35:37.346 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543/remote/smithi186/crash 2024-02-20T09:35:37.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:37.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:37.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:37.231+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:37.232169+0000) 2024-02-20T09:35:37.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:37.215+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:37.215979+0000) 2024-02-20T09:35:38.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:38.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:38.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:38.117+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:38.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:38.118+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:38.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:38.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:38.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:38.037+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:38.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:38.038+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:37.923+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:37.924406+0000) 2024-02-20T09:35:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:38.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.029+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:38.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:38.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:38.233+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:38.234284+0000) 2024-02-20T09:35:38.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:38.217+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:38.218617+0000) 2024-02-20T09:35:39.159 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:38.840+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69293 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:39.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:38.835+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:38.836208+0000) 2024-02-20T09:35:39.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:39.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:39.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:39.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:39.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:39.015+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:39.160 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:39.016+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.836+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:38.837817+0000) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.925+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:38.926435+0000) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.994+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:38.995+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:39.161 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:38.835+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:38.836364+0000) 2024-02-20T09:35:39.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:39.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:39.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:39.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:39.162 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:39.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:39.471 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:39.219+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:39.220663+0000) 2024-02-20T09:35:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:39.158+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:39.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:39.235+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:39.236350+0000) 2024-02-20T09:35:40.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.021+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:40.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.021+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:40.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:40.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:40.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:40.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:40.157+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:40.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:40.158+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:39.928+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:39.929599+0000) 2024-02-20T09:35:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:40.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:40.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:40.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:40.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:40.028+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:40.416 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:40.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.222+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:40.223140+0000) 2024-02-20T09:35:40.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:40.238+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:40.238957+0000) 2024-02-20T09:35:41.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:41.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:41.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:41.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:41.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:41.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:41.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:41.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:40.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:40.931+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:40.932901+0000) 2024-02-20T09:35:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:41.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:41.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:41.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:41.012+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:41.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:41.013+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:41.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:41.240+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:41.241368+0000) 2024-02-20T09:35:41.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:41.224+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:41.225210+0000) 2024-02-20T09:35:42.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:42.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:42.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:42.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:42.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:42.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:41.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:41.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:42.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:41.982+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:42.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:41.983+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:41.935+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:41.936318+0000) 2024-02-20T09:35:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:42.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:42.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:42.020+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:42.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:42.022+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:42.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:42.226+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:42.227292+0000) 2024-02-20T09:35:42.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:42.242+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:42.243457+0000) 2024-02-20T09:35:43.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:43.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:43.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:43.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:43.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:43.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:42.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:43.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:42.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:42.963+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:43.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:42.964+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:42.937+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:42.938787+0000) 2024-02-20T09:35:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:43.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:43.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:43.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:43.015+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:43.016+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:43.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:43.228+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:43.229522+0000) 2024-02-20T09:35:43.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:43.244+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:43.245910+0000) 2024-02-20T09:35:44.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:43.848+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69332 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:43.845+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:43.846681+0000) 2024-02-20T09:35:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:43.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:43.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:43.993+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:44.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:43.994+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:43.845+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:43.846360+0000) 2024-02-20T09:35:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:43.939+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:43.940902+0000) 2024-02-20T09:35:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:44.061+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:44.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:44.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:44.062+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:44.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:44.063+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:44.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:43.845+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:43.846077+0000) 2024-02-20T09:35:44.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:44.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:44.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:44.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:44.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:44.043+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:44.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:44.044+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:44.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:44.230+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:44.231634+0000) 2024-02-20T09:35:44.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:44.246+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:44.247234+0000) 2024-02-20T09:35:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:45.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:45.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:45.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:45.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:45.034+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:44.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:44.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:45.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:44.946+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:45.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:44.947+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:44.941+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:44.942898+0000) 2024-02-20T09:35:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:45.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:45.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:45.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:45.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:45.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:45.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:45.232+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:45.233763+0000) 2024-02-20T09:35:45.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:45.248+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:45.249639+0000) 2024-02-20T09:35:46.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:46.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:46.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:46.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:46.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:46.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:45.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:46.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:45.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:45.918+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:46.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:45.919+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:45.945+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:45.946485+0000) 2024-02-20T09:35:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:46.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:46.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:46.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:46.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:46.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:46.006+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:46.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:46.250+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:46.251874+0000) 2024-02-20T09:35:46.575 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:46.236+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:46.237011+0000) 2024-02-20T09:35:47.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:47.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:47.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:47.026+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:47.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:47.028+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:46.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:47.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:46.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:46.925+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:47.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:46.926+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:46.948+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:46.949852+0000) 2024-02-20T09:35:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:47.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.022+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:47.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.023+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:47.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:47.253+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:47.254019+0000) 2024-02-20T09:35:47.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:47.238+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:47.239198+0000) 2024-02-20T09:35:48.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:47.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:47.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:47.932+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:48.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:47.933+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:48.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:48.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:48.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:48.070+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:48.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:48.071+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:48.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.952+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:47.953550+0000) 2024-02-20T09:35:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:48.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:47.982+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:48.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:48.240+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:48.241588+0000) 2024-02-20T09:35:48.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:48.255+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:48.256032+0000) 2024-02-20T09:35:49.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:48.857+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69372 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:48.854+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:48.855020+0000) 2024-02-20T09:35:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:48.953+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:48.954906+0000) 2024-02-20T09:35:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:49.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:49.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:49.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:49.030+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:49.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:49.030+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:48.853+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:48.854716+0000) 2024-02-20T09:35:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:49.041+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:49.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:49.042+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:48.855+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:48.856745+0000) 2024-02-20T09:35:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:48.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:48.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:48.942+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:49.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:48.943+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:49.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:35:49.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:49.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:49.257+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:49.258229+0000) 2024-02-20T09:35:49.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:49.241+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:49.242883+0000) 2024-02-20T09:35:50.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:49.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:49.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:49.983+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:50.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:49.984+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:50.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:50.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:50.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:50.009+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:50.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:50.010+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:49.956+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:49.957509+0000) 2024-02-20T09:35:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.018+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.019+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:50.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:50.259+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:50.260396+0000) 2024-02-20T09:35:50.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:50.245+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:50.246322+0000) 2024-02-20T09:35:51.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.958+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:50.959739+0000) 2024-02-20T09:35:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.995+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:51.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:50.996+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:51.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:51.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:51.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:51.010+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:51.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:51.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:51.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.262+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:51.263746+0000) 2024-02-20T09:35:51.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:51.248+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:51.249715+0000) 2024-02-20T09:35:52.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:52.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:52.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:52.011+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:52.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:52.011+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:51.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:52.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:51.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:51.959+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:51.960+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:51.962+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:51.963053+0000) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.977+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:52.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:51.978+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:52.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:35:52.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:52.266+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:52.267098+0000) 2024-02-20T09:35:52.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:52.251+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:52.252334+0000) 2024-02-20T09:35:53.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:53.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:53.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:53.024+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:53.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:53.025+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:53.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:53.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:53.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:53.054+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:53.055+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:52.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:52.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:52.916+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:52.917+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:53.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:52.964+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:52.965774+0000) 2024-02-20T09:35:53.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:53.253+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:53.254757+0000) 2024-02-20T09:35:53.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:53.269+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:53.270324+0000) 2024-02-20T09:35:54.221 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:53.866+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69411 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:53.861+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:53.862719+0000) 2024-02-20T09:35:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:54.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:54.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:54.030+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:54.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:54.030+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:53.862+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:53.863456+0000) 2024-02-20T09:35:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:54.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:54.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:54.063+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:54.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:54.064+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:54.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:53.862+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:53.863817+0000) 2024-02-20T09:35:54.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:53.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:54.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:53.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:54.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:53.876+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:54.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:53.877+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:54.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:53.966+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:53.967881+0000) 2024-02-20T09:35:54.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:54.255+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:54.256852+0000) 2024-02-20T09:35:54.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:54.271+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:54.272428+0000) 2024-02-20T09:35:55.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:55.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:55.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:55.078+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:55.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:55.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:54.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:55.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:54.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:54.875+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:54.876+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:54.970+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:54.971469+0000) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:55.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:55.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:55.076+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:55.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:55.077+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:55.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:55.274+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:55.275015+0000) 2024-02-20T09:35:55.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:55.258+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:55.259312+0000) 2024-02-20T09:35:56.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:56.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:56.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:56.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:56.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:56.079+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:56.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:56.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:56.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:56.087+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:56.087+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:55.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:55.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:55.898+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:56.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:55.900+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:56.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:55.973+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:55.974019+0000) 2024-02-20T09:35:56.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:56.277+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:56.278348+0000) 2024-02-20T09:35:56.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:56.260+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:56.261342+0000) 2024-02-20T09:35:57.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:57.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:57.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:57.126+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:57.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:57.127+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:57.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:57.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:57.060+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:57.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:57.061+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:56.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:57.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:56.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:56.942+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:56.943+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:57.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:56.976+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:56.977368+0000) 2024-02-20T09:35:57.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:57.280+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:57.281789+0000) 2024-02-20T09:35:57.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:57.262+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:57.263334+0000) 2024-02-20T09:35:58.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:58.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:58.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:58.141+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:58.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:58.142+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:58.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:58.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:58.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:58.048+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:58.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:58.049+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:57.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:57.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:57.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:58.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:57.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:58.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:57.979+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:57.980841+0000) 2024-02-20T09:35:58.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:58.283+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:58.283929+0000) 2024-02-20T09:35:58.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:58.265+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:58.266153+0000) 2024-02-20T09:35:59.191 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:35:58.877+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69451 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:35:59.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:58.871+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:58.872609+0000) 2024-02-20T09:35:59.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:59.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:59.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:59.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:59.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:59.089+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:35:59.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:59.090+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:35:59.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:58.871+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:58.872899+0000) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:58.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:58.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:58.931+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:58.932+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:58.981+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:58.982395+0000) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:58.871+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:58.872388+0000) 2024-02-20T09:35:59.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:59.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:59.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:59.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:59.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:59.189+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:35:59.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:59.190+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:35:59.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:35:59.284+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:59.285057+0000) 2024-02-20T09:35:59.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:35:59.266+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:59.267864+0000) 2024-02-20T09:36:00.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:00.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:00.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:00.091+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:00.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:00.092+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:59.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:59.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:00.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:59.946+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:59.947+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:00.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:35:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:35:59.984+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:35:59.985035+0000) 2024-02-20T09:36:00.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:00.268+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:00.270313+0000) 2024-02-20T09:36:00.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:00.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:00.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:00.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:00.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:00.238+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:00.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:00.239+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:00.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:00.285+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:00.287714+0000) 2024-02-20T09:36:01.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:01.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:01.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:01.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:01.101+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:01.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:01.102+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:01.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:00.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:01.225 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:00.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:01.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:00.924+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:01.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:00.925+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:01.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:00.985+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:00.987254+0000) 2024-02-20T09:36:01.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:01.271+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:01.273377+0000) 2024-02-20T09:36:01.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:01.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:01.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:01.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:01.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:01.244+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:01.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:01.245+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:01.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:01.289+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:01.291063+0000) 2024-02-20T09:36:02.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:02.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:02.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:02.126+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:02.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:02.127+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:01.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:01.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:02.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:01.919+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:01.920+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:02.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:01.987+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:01.989898+0000) 2024-02-20T09:36:02.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:02.274+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:02.276825+0000) 2024-02-20T09:36:02.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:02.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:02.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:02.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:02.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:02.247+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:02.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:02.248+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:02.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:02.292+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:02.294455+0000) 2024-02-20T09:36:03.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:03.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:03.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:03.171+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:03.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:03.172+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:02.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:02.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:03.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:02.941+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:02.941+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:03.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:02.991+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:02.992572+0000) 2024-02-20T09:36:03.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:03.276+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:03.278881+0000) 2024-02-20T09:36:03.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:03.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:03.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:03.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:03.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:03.230+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:03.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:03.230+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:03.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:03.295+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:03.296673+0000) 2024-02-20T09:36:04.190 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:03.885+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69490 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:04.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:03.882+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:03.883740+0000) 2024-02-20T09:36:04.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:03.882+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:03.883223+0000) 2024-02-20T09:36:04.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:03.881+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:03.882914+0000) 2024-02-20T09:36:04.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:03.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:04.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:03.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:04.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:03.957+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:04.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:03.957+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:04.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:03.993+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:03.994420+0000) 2024-02-20T09:36:04.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:04.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:04.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:04.219+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:04.221+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:04.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:04.297+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:04.297956+0000) 2024-02-20T09:36:04.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:04.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:04.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:04.187+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:04.188+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:04.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:04.280+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:04.281181+0000) 2024-02-20T09:36:04.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:36:04.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:05.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:05.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:05.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:05.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:05.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:05.152+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:05.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:05.153+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:05.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:04.996+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:04.996944+0000) 2024-02-20T09:36:05.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:05.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:05.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.001+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:05.261 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.002+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:05.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:05.282+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:05.283457+0000) 2024-02-20T09:36:05.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:05.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:05.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:05.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:05.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:05.258+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:05.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:05.259+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:05.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:05.299+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:05.300878+0000) 2024-02-20T09:36:06.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:06.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:06.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:06.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:06.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:06.170+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:06.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:06.171+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:06.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:06.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:06.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.982+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:06.252 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.983+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:06.252 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:05.998+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:05.999454+0000) 2024-02-20T09:36:06.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:06.286+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:06.287043+0000) 2024-02-20T09:36:06.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:06.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:06.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:06.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:06.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:06.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:06.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:06.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:06.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:06.302+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:06.303154+0000) 2024-02-20T09:36:07.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:06.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:07.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:06.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:07.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:06.993+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:07.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:06.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:07.211 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:07.000+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:07.001650+0000) 2024-02-20T09:36:07.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:07.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:07.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:07.293+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:07.294+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:07.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:07.304+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:07.305515+0000) 2024-02-20T09:36:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:07.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:07.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:07.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:07.208+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:07.209+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:07.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:07.288+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:07.289711+0000) 2024-02-20T09:36:07.552 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:36:08.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:08.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:08.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:08.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:08.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:08.248+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:08.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:08.249+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:08.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:08.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:08.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:08.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:08.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:08.211+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:08.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:08.212+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:08.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:08.004+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:08.005027+0000) 2024-02-20T09:36:08.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:08.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:08.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:08.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:08.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:08.037+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:08.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:08.037+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:08.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:08.307+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:08.308164+0000) 2024-02-20T09:36:08.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:08.291+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:08.292259+0000) 2024-02-20T09:36:08.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:08.894+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69530 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:08.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:08.890+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:08.891488+0000) 2024-02-20T09:36:08.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:08.891+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:08.892243+0000) 2024-02-20T09:36:08.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:08.890+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:08.891283+0000) 2024-02-20T09:36:09.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:09.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:09.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:09.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:09.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:09.210+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:09.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:09.211+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:09.297 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:09.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:09.297 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:09.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:09.297 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:09.245+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:09.297 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:09.246+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:09.006+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:09.006953+0000) 2024-02-20T09:36:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:09.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:09.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:09.040+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:09.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:09.041+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:09.694 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:09.308+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:09.309719+0000) 2024-02-20T09:36:09.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:09.293+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:09.294152+0000) 2024-02-20T09:36:10.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:10.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:10.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:10.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:10.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:10.177+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:10.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:10.177+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:10.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:10.008+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:10.009590+0000) 2024-02-20T09:36:10.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:10.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:10.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:10.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:10.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:10.021+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:10.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:10.021+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:10.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:36:10.554 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:10.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:36:10.620 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:36:10.620 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-20T09:36:10.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-20T09:36:10.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:10.312+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:10.313103+0000) 2024-02-20T09:36:10.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:10.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:10.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:10.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:10.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:10.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:10.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:10.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:10.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:10.296+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:10.297668+0000) 2024-02-20T09:36:10.659 INFO:tasks.cephadm:Compressing logs... 2024-02-20T09:36:10.659 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T09:36:10.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:36:10.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:10.727 INFO:teuthology.orchestra.run.smithi055.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi055.ptjwai.log 2024-02-20T09:36:10.728 INFO:teuthology.orchestra.run.smithi055.stderr:: No such file or directory 2024-02-20T09:36:10.728 INFO:teuthology.orchestra.run.smithi055.stderr:gzip/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi055.ptjwai.log: -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-20T09:36:10.730 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi055.stjper.log 2024-02-20T09:36:10.730 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log 2024-02-20T09:36:10.730 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi055.stjper.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi055.sinapd.log 2024-02-20T09:36:10.730 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log 2024-02-20T09:36:10.730 INFO:teuthology.orchestra.run.smithi055.stderr: 89.9% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-20T09:36:10.731 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi055.sinapd.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi055.log 2024-02-20T09:36:10.732 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-20T09:36:10.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi055.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.log 2024-02-20T09:36:10.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log 2024-02-20T09:36:10.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.log: 81.6% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-20T09:36:10.741 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-20T09:36:10.742 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T09:36:10.755 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T09:36:10.773 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T09:36:10.782 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cephadm.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T09:36:10.824 INFO:teuthology.orchestra.run.smithi055.stderr: 95.4% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi055.ptjwai.log.gz 2024-02-20T09:36:11.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:11.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:11.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:11.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:11.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:11.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:11.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:11.204+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:11.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:11.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:11.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:11.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:11.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:11.291+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:11.012+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:11.012958+0000) 2024-02-20T09:36:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:11.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:11.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:11.031+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:11.034+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:11.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:11.318+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:11.319001+0000) 2024-02-20T09:36:11.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:11.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:11.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:11.300+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:11.300956+0000) 2024-02-20T09:36:12.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:12.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:12.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:12.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:12.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:12.242+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:12.302 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:12.243+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:12.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:12.017+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:12.018161+0000) 2024-02-20T09:36:12.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:12.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:12.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:12.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:12.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:12.032+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:12.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:12.032+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:12.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:12.339+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:12.340782+0000) 2024-02-20T09:36:12.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:12.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:12.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:12.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:12.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:12.298+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:12.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:12.298+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:12.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:12.304+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:12.304952+0000) 2024-02-20T09:36:12.756 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi055.stjper.log: file size changed while zipping 2024-02-20T09:36:12.756 INFO:teuthology.orchestra.run.smithi055.stderr: 90.0% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi055.stjper.log.gz 2024-02-20T09:36:13.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:13.019+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:13.020905+0000) 2024-02-20T09:36:13.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:13.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:13.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:13.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:13.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:13.080+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:13.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:13.081+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:13.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:13.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:13.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:13.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:13.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:13.270+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:13.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:13.271+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:13.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:13.343+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:13.343968+0000) 2024-02-20T09:36:13.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:13.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:13.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:13.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:13.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:13.283+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:13.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:13.284+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:13.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:13.307+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:13.307975+0000) 2024-02-20T09:36:13.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:36:14.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:13.934+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:13.935867+0000) 2024-02-20T09:36:14.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:13.911+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:13.912877+0000) 2024-02-20T09:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:13.915+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:13.916307+0000) 2024-02-20T09:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:14.021+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:14.022756+0000) 2024-02-20T09:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:14.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:14.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:14.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:14.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:14.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:14.224 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:13.914+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69569 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:14.706 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:14.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:14.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:14.305+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:14.306+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:14.707 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:14.344+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:14.345684+0000) 2024-02-20T09:36:14.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:14.310+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:14.310992+0000) 2024-02-20T09:36:14.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:14.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:14.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:14.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:14.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:14.324+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:14.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:14.325+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:15.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:15.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:15.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:15.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:15.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:15.262+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:15.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:15.262+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:15.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:15.025+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:15.025958+0000) 2024-02-20T09:36:15.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:15.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:15.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:15.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:15.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:15.105+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:15.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:15.106+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:15.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:15.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:15.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:15.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:15.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:15.293+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:15.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:15.294+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:15.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:15.312+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:15.313526+0000) 2024-02-20T09:36:15.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:15.347+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:15.348646+0000) 2024-02-20T09:36:16.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:16.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:16.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:16.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:16.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:16.299+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:16.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:16.299+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:16.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:16.028+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:16.028955+0000) 2024-02-20T09:36:16.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:16.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:16.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:16.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:16.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:16.069+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:16.305 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:16.070+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:16.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:16.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:16.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:16.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:16.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:16.308+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:16.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:16.309+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:16.710 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:16.350+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:16.351784+0000) 2024-02-20T09:36:16.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:16.316+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:16.316986+0000) 2024-02-20T09:36:17.327 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:17.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:17.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:17.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:17.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:17.260+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:17.328 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:17.261+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:17.031+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:17.031956+0000) 2024-02-20T09:36:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:17.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:17.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:17.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:17.329 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:17.121+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:17.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:17.353+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:17.354302+0000) 2024-02-20T09:36:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:17.322+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:17.323026+0000) 2024-02-20T09:36:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:17.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:17.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:17.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:17.347+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:17.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:17.347+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:18.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:18.214+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:18.329 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:18.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:18.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:18.215+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:18.330 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:18.216+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:18.330 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:18.324+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:18.325724+0000) 2024-02-20T09:36:18.330 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:18.033+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:18.034651+0000) 2024-02-20T09:36:18.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:18.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:18.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:18.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:18.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:18.082+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:18.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:18.083+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:18.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:18.356+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:18.357027+0000) 2024-02-20T09:36:18.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:18.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:18.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:18.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:18.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:18.372+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:18.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:18.373+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:18.966 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:18.923+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69609 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:18.966 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:18.933+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:18.934449+0000) 2024-02-20T09:36:18.967 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:18.928+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:18.929193+0000) 2024-02-20T09:36:18.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:18.945+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:18.946868+0000) 2024-02-20T09:36:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:19.036+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:19.036956+0000) 2024-02-20T09:36:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:19.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:19.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:19.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:19.098+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:19.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:19.099+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:19.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:19.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:19.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:19.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:19.224 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:19.203+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:19.225 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:19.203+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:19.326+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:19.327457+0000) 2024-02-20T09:36:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:19.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:19.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:19.368+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:19.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:19.371+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:19.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:19 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:19.357+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:19.358793+0000) 2024-02-20T09:36:20.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:20.038+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:20.039528+0000) 2024-02-20T09:36:20.332 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:20.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:20.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:20.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:20.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:20.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:20.333 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:20.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:20.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:20.329+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:20.330042+0000) 2024-02-20T09:36:20.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:20.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:20.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:20.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:20.334 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:20.155+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:20.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:20.156+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:20.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:20.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:20.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:20.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:20.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:20.355+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:20.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:20.355+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:20.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:20 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:20.360+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:20.361342+0000) 2024-02-20T09:36:21.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:21.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:21.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:21.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:21.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:21.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:21.335 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:21.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:21.336 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:21.332+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:21.333254+0000) 2024-02-20T09:36:21.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:21.042+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:21.042957+0000) 2024-02-20T09:36:21.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:21.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:21.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:21.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:21.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:21.083+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:21.337 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:21.084+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:21.363+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:21.364446+0000) 2024-02-20T09:36:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:21.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:21.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:21.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:21.401+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:21.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:21 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:21.402+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:22.338 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:22.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:22.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:22.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:22.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:22.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:22.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:22.137+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:22.339 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:22.044+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:22.045597+0000) 2024-02-20T09:36:22.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:22.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:22.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:22.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:22.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:22.119+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:22.340 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:22.122+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:22.341 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:22.336+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:22.336962+0000) 2024-02-20T09:36:22.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:22.367+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:22.367968+0000) 2024-02-20T09:36:22.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:22.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:22.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:22.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:22.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:22.409+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:22.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:22 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:22.410+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:22.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:36:22.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:23.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:23.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:23.341 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:23.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:23.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:23.147+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:23.342 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:23.148+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:23.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:23.046+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:23.047781+0000) 2024-02-20T09:36:23.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:23.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:23.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:23.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:23.342 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:23.123+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:23.343 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:23.124+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:23.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:23.368+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:23.369865+0000) 2024-02-20T09:36:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:23.339+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:23.340176+0000) 2024-02-20T09:36:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:23.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:23.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:23.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:23.415+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:23.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:23.416+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:23.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:23.937+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69648 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:23.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:23.932+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:23.933581+0000) 2024-02-20T09:36:23.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:23.944+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:23.945327+0000) 2024-02-20T09:36:23.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:23 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:23.937+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:23.938346+0000) 2024-02-20T09:36:24.344 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:24.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:24.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:24.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:24.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:24.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:24.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:24.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:24.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:24.049+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:24.049952+0000) 2024-02-20T09:36:24.345 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:24.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:24.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:24.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:24.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:24.139+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:24.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:24.141+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:24.346 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:24.341+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:24.342912+0000) 2024-02-20T09:36:24.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:24.370+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:24.371676+0000) 2024-02-20T09:36:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:24.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:24.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:24.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:24.464+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:24.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:24 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:24.465+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:25.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:25.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:25.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:25.114+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:25.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:25.115+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:25.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:25.052+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:25.053537+0000) 2024-02-20T09:36:25.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:25.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:25.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:25.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:25.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:25.112+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:25.351 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:25.113+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:25.534 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-20T09:36:25.535 INFO:teuthology.orchestra.run.smithi055.stderr: 95.1% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log.gz 2024-02-20T09:36:25.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:25.373+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:25.374644+0000) 2024-02-20T09:36:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:25.347+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:25.348260+0000) 2024-02-20T09:36:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:25.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:25.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:25.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:25.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:25 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:25.480+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:25.824 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:36:26.346 INFO:teuthology.orchestra.run.smithi055.stderr: 92.2% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi055.sinapd.log.gz 2024-02-20T09:36:26.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:26.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:26.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:26.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:26.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:26.129+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:26.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:26.130+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:26.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:26.351+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:26.352193+0000) 2024-02-20T09:36:26.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:26.055+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:26.056210+0000) 2024-02-20T09:36:26.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:26.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:26.355 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:26.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:26.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:26.077+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:26.356 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:26.078+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:26.389 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-20T09:36:26.389 INFO:teuthology.orchestra.run.smithi055.stderr: 95.1% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log.gz 2024-02-20T09:36:26.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:26.376+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:26.377751+0000) 2024-02-20T09:36:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:26.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:26.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:26.477+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:26.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:26 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:26.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:27.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:27.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:27.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:27.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:27.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:27.164+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:27.356 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:27.165+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:27.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:27.058+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:27.059229+0000) 2024-02-20T09:36:27.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:27.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:27.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:27.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:27.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:27.075+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:27.357 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:27.076+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:27.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:27.380+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:27.381301+0000) 2024-02-20T09:36:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:27.353+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:27.354803+0000) 2024-02-20T09:36:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:27.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:27.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:27.501+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:27.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:27 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:27.502+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:28.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:28.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:28.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:28.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:28.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:28.128+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:28.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:28.130+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:28.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:28.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:28.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:28.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:28.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:28.052+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:28.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:28.053+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:28.360 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:28.061+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:28.062471+0000) 2024-02-20T09:36:28.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:28.383+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:28.383983+0000) 2024-02-20T09:36:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:28.356+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:28.357893+0000) 2024-02-20T09:36:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:28.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:28.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:28.486+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:28.487+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:29.016 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:28.948+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69688 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:29.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:28.944+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:28.944946+0000) 2024-02-20T09:36:29.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:28.945+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:28.946023+0000) 2024-02-20T09:36:29.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:28 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:28.944+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:28.945196+0000) 2024-02-20T09:36:29.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:29.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:29.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:29.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:29.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:29.014+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:29.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:29.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:29.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:29.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:29.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:29.163+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:29.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:29.164+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:29.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:29.015+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:29.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:29.063+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:29.064552+0000) 2024-02-20T09:36:29.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:29.385+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:29.386249+0000) 2024-02-20T09:36:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:29.359+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:29.360818+0000) 2024-02-20T09:36:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:29.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:29.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:29.467+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:29.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:29 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:29.468+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:30.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:30.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:30.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:30.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:30.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:30.154+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:30.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:30.155+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:30.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:30.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:30.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:30.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:30.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:30.055+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:30.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:30.056+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:30.366 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:30.066+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:30.067848+0000) 2024-02-20T09:36:30.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:30.388+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:30.389433+0000) 2024-02-20T09:36:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:30.362+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:30.363105+0000) 2024-02-20T09:36:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:30.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:30.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:30.463+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:30.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:30 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:30.464+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:31.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:31.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:31.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:31.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:31.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:31.119+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:31.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:31.120+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:31.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:31.070+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:31.071335+0000) 2024-02-20T09:36:31.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:31.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:31.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:31.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:31.369 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:31.072+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:31.370 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:31.074+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:31.370 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:31.365+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:31.366503+0000) 2024-02-20T09:36:31.439 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi055.log: file size changed while zipping 2024-02-20T09:36:31.440 INFO:teuthology.orchestra.run.smithi055.stderr: 98.1% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi055.log.gz 2024-02-20T09:36:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:31.391+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:31.392833+0000) 2024-02-20T09:36:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:31.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:31.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:31.420+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:31.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:31 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:31.422+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:32.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:32.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:32.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:32.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:32.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:32.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:32.371 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:32.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:32.371 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:32.074+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:32.075081+0000) 2024-02-20T09:36:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:32.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:32.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:32.100+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:32.372 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:32.101+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:32.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:32.395+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:32.396380+0000) 2024-02-20T09:36:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:32.368+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:32.369493+0000) 2024-02-20T09:36:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:32.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:32.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:32.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:32.375+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:32.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:32 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:32.376+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:32.906 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-20T09:36:32.906 INFO:teuthology.orchestra.run.smithi055.stderr: 96.6% -- replaced with /var/log/ceph/9a1162a4-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log.gz 2024-02-20T09:36:32.908 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-02-20T09:36:32.909 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m22.194s 2024-02-20T09:36:32.909 INFO:teuthology.orchestra.run.smithi055.stderr:user 1m28.439s 2024-02-20T09:36:32.909 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m1.852s 2024-02-20T09:36:33.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:33.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:33.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:33.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:33.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:33.134+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:33.374 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:33.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:33.374 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:33.076+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:33.077570+0000) 2024-02-20T09:36:33.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:33.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:33.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:33.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:33.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:33.101+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:33.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:33.102+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:33.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:33.397+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:33.398905+0000) 2024-02-20T09:36:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:33.371+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:33.372710+0000) 2024-02-20T09:36:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:33.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:33.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:33.389+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:33.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:33.390+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:34.066 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:33.956+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69727 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:34.066 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:33.952+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:33.953925+0000) 2024-02-20T09:36:34.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:33.953+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:33.954104+0000) 2024-02-20T09:36:34.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:33 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:33.953+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:33.954768+0000) 2024-02-20T09:36:34.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:34.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:34.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:34.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:34.067 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:34.064+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:34.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:34.065+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:34.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:34.078+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:34.079606+0000) 2024-02-20T09:36:34.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:34.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:34.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:34.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:34.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:34.136+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:34.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:34.136+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:34.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:34.399+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:34.400440+0000) 2024-02-20T09:36:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:34.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:34.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:34.362+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:34.363+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:34.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:34 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:34.374+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:34.375066+0000) 2024-02-20T09:36:35.379 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:35.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:35.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:35.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:35.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:35.160+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:35.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:35.161+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:35.380 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:35.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:35.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.027+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:35.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.028+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:35.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.081+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:35.082114+0000) 2024-02-20T09:36:35.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:35.402+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:35.402954+0000) 2024-02-20T09:36:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:35.377+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:35.378507+0000) 2024-02-20T09:36:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:35.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:35.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:35.407+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:35.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:35.408+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:36.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:36.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:36.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:36.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:36.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:36.133+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:36.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:36.134+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:36.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:36.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:36.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.992+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:36.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:35 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:35.993+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:36.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:36.084+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:36.085106+0000) 2024-02-20T09:36:36.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:36.404+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:36.405468+0000) 2024-02-20T09:36:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:36.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:36.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:36.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:36.381+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:36.382182+0000) 2024-02-20T09:36:36.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:36 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:36.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:37.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:37.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:37.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:37.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:37.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:37.097+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:37.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:37.098+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:37.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:37.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:37.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:37.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:37.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:37.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:37.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:37.007+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:37.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:37.087+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:37.088662+0000) 2024-02-20T09:36:37.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:37.406+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:37.407583+0000) 2024-02-20T09:36:37.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:37.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:37.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:37.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:37.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:37.377+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:37.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:37.378+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:37.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:37 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:37.383+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:37.384718+0000) 2024-02-20T09:36:37.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:36:37.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:38.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:38.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:38.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:38.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:38.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:38.091+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:38.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:38.092+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:38.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:38.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:38.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:38.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:38.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:38.016+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:38.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:38.017+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:38.389 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:38.089+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:38.090895+0000) 2024-02-20T09:36:38.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:38.408+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:38.409884+0000) 2024-02-20T09:36:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:38.386+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:38.386967+0000) 2024-02-20T09:36:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:38.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:38.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:38.425+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:38.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:38.426+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:39.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:38.962+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:38.963788+0000) 2024-02-20T09:36:39.008 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:38.964+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69763 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:39.008 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:38.960+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:38.961790+0000) 2024-02-20T09:36:39.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:38 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:38.961+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:38.962315+0000) 2024-02-20T09:36:39.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:39.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:39.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:39.382 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:39.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:39.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:39.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:39.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:39.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:39.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:39.111+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:39.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:39.112+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:39.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:39.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.005+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:39.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.006+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:39.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.091+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:39.092596+0000) 2024-02-20T09:36:39.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:39.411+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:39.412161+0000) 2024-02-20T09:36:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:39.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:39.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:39.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:39.388+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:39.389174+0000) 2024-02-20T09:36:40.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:40.092 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:40.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.980+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:40.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:39 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:39.981+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:40.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:40.093+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:40.094892+0000) 2024-02-20T09:36:40.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:40.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:40.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:40.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:40.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:40.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:40.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:40.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:40.639 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:40.414+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:40.414968+0000) 2024-02-20T09:36:40.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:40.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:40.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:40.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:40.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:40.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:40.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:40.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:40.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:40 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:40.390+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:40.391731+0000) 2024-02-20T09:36:40.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:36:41.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:41.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:41.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:41.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:41.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:41.090+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:41.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:41.091+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:41.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:41.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:41.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:41.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:41.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:41.025+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:41.390 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:41.026+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:41.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:41.097+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:41.098280+0000) 2024-02-20T09:36:41.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:41.417+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:41.418390+0000) 2024-02-20T09:36:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:41.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:41.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:41.387+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:41.388+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:41.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:41 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:41.393+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:41.394047+0000) 2024-02-20T09:36:42.361 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:42.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:42.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:42.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:42.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:42.057+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:42.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:42.058+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:42.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:42.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:42.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:42.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:42.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:42.038+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:42.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:42.039+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:42.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:42.100+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:42.101134+0000) 2024-02-20T09:36:42.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:42.419+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:42.420650+0000) 2024-02-20T09:36:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:42.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:42.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:42.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:42.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:42.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:42.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:42 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:42.395+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:42.396764+0000) 2024-02-20T09:36:43.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:43.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:43.321 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:43.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:43.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:43.067+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:43.322 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:43.068+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:43.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:43.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:43.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:43.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:43.322 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:43.081+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:43.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:43.082+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:43.323 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:43.103+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:43.104799+0000) 2024-02-20T09:36:43.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:43.422+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:43.423143+0000) 2024-02-20T09:36:43.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:43.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:43.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:43.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:43.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:43.319+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:43.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:43.320+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:43.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:43.399+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:43.400102+0000) 2024-02-20T09:36:43.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:36:43.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:43.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:36:43.964 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:36:43.964 INFO:tasks.cephadm:Removing cluster... 2024-02-20T09:36:43.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 --force 2024-02-20T09:36:44.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:43.970+0000 7f0db93ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:43.971176+0000) 2024-02-20T09:36:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:43.974+0000 7fcd3aa04700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:43.975903+0000) 2024-02-20T09:36:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:44.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:44.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:44.108+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:44.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:44.109+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:43.971+0000 7f150f571700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:43.972732+0000) 2024-02-20T09:36:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:44.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:44.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:44.066+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:44.067+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:44.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:44.105+0000 7f150018c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:44.106332+0000) 2024-02-20T09:36:44.224 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:43 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:43.975+0000 7fc099d3e700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 69802 slow ops, oldest is monmgrreport(gid 14216, 0 checks, 1 progress events) 2024-02-20T09:36:44.547 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:44 smithi055 systemd[1]: Stopping Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:36:44.547 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:44.424+0000 7fcd2c621700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:44.425028+0000) 2024-02-20T09:36:44.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:44.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:44.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:44.348+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:44.349+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:44.401+0000 7f0daa7c8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:36:44.402421+0000) 2024-02-20T09:36:44.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:44.584+0000 7fc09cd44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi055 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:36:44.972 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:44 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055[29250]: 2024-02-20T09:36:44.584+0000 7fc09cd44700 -1 mon.smithi055@0(probing) e2 *** Got Signal Terminated *** 2024-02-20T09:36:45.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:45.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:45.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:45.122+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:45.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:45.123+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:45.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:45.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:45.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:45.322+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:45.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:45.322+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:45.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:45.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:45.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:45.113+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:45.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:45 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:45.114+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:46.027 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:45 smithi055 podman[93701]: 2024-02-20 09:36:45.72238732 +0000 UTC m=+1.272700746 container died a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-20T09:36:46.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:46 smithi055 podman[93701]: 2024-02-20 09:36:46.241552376 +0000 UTC m=+1.791865800 container cleanup a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, io.buildah.version=1.33.5) 2024-02-20T09:36:46.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:46 smithi055 bash[93701]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055 2024-02-20T09:36:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:46.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:46.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:46.093+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:46.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:46.094+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:46.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:46.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:46.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:46.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:46.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:46.079+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:46.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:46.080+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:46.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:46 smithi055 podman[93715]: 2024-02-20 09:36:46.493693946 +0000 UTC m=+0.772509239 container remove a759d641dbe8efd43987e30a67d5396488d3602615479886d891016daf52f8e9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-mon-smithi055, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-20T09:36:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:46.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:46.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:46.346+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:46.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:46 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:46.346+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:47.056 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:46 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service: Deactivated successfully. 2024-02-20T09:36:47.056 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:46 smithi055 systemd[1]: Stopped Ceph mon.smithi055 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:36:47.056 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Feb 20 09:36:46 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@mon.smithi055.service: Consumed 1h 17min 9.266s CPU time. 2024-02-20T09:36:47.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:47.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:47.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:47.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:47.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:47.093+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:47.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:47.094+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:47.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:47.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:47.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:47.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:47.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:47.053+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:47.373 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:47.054+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:47.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:47.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:47.369+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:47.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:47 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:47.370+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:48.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:48.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:48.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:48.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:48.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:48.085+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:48.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:48.086+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:48.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:48.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:48.383 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:48.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:48.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:48.045+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:48.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:48.045+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:48.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:48.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:48.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:48.380+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:48.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:48 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:48.381+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:49.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:49.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:49.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:49.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:49.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:49.063+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:49.361 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:49.064+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:49.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:49.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:49.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:49.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:49.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:49.049+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:49.362 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:49.050+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:49.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:49.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:49.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:49.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:49.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:49.359+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:49.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:49 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:49.360+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:50.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:50.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:50.345 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:50.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:50.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:50.019+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:50.346 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:50.020+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:50.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:50.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:50.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:50.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:50.346 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:50.095+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:50.347 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:50.096+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:50.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:50.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:50.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:50.342+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:50.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:50 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:50.343+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:51.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:51.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:51.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:51.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:51.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:51.034+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:51.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:51.035+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:51.364 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:51.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:51.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:51.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:51.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:51.086+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:51.365 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:51.087+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:51.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:51.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:51.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:51.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:51.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:51.360+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:51.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:51 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:51.361+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:52.319 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:52.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:52.320 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:52.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:52.320 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:52.065+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:52.320 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:52.065+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:52.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:52.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:52.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:52.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:52.320 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:52.107+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:52.321 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:52.108+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:52.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:52.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:52.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:52.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:52.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:52.390+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:52.601 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:52 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:52.391+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:53.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:53.060+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:53.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:53.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:53.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:53.061+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:53.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:53.061+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:53.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:53.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:53.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:53.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:53.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:53.154+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:53.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:53.155+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:53.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:53.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:53.410+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:53.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:53 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:53.411+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:54.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:54.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:54.381 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:54.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:54.170+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:54.382 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:54.171+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:54.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:54.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:54.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:54.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:54.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:54.058+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:54.382 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:54.059+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:54.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:54.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:54.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:54.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:54.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:54.378+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:54.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:54 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:54.379+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:55.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:55.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:55.146 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:55.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:55.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:55.080+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:55.147 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:55.081+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:55.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:55.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:55.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:55.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:55.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:55.144+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:55.406 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:55.144+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:55.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:55.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:55.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:55.416+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:55.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:55 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:55.416+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:56.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:56.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:56.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.052+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:56.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.053+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:56.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 systemd[1]: Stopping Ceph osd.0 for 9a1162a4-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:36:56.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.364+0000 7fcd381ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:36:56.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.364+0000 7fcd381ff700 -1 osd.0 46 *** Got signal Terminated *** 2024-02-20T09:36:56.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:56.364+0000 7fcd381ff700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-20T09:36:56.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:56.192+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:56.453 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:56.192+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:56.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:56.192+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:56.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:56.192+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:56.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:56.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:56.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:56.449+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:56.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:56 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:56.450+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:57.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:57.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:57.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:57.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:57.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:57.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:57.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:57.038+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:57.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:57.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:57.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:57.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:57.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:57.233+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:57.430 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:57.234+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:57.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:57.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:57.426+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:57.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:57 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:57.427+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:58.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:58.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:58.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:58.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:58.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:58.037+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:58.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:58.037+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:58.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:58.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:58.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:58.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:58.477 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:58.250+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:58.478 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:58.251+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:58.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:58.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:58.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:58.471+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:58.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:58 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:58.472+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.021+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:36:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.022+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:36:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:59.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:59.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:59.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:36:59.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:36:59.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:36:59.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:59.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:59.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:59.287+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:36:59.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:36:59.288+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:00.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:37:00.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:37:00.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.982+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:37:00.333 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:36:59 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:36:59.983+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:00.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:00.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:00.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:00.458+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:00.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:00.459+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:00.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:00.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:00.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:00.331+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:00.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:00.331+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:01.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:37:00.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:37:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:37:00.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:37:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:37:00.936+0000 7fcd36000700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:42.325924+0000 (oldest deadline 2024-02-20T09:35:02.325924+0000) 2024-02-20T09:37:01.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:00 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0[48248]: 2024-02-20T09:37:00.936+0000 7fcd36000700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1928 3.6 3:60bfd03c:::100000020a0.00000000:head [write 0~777 in=777b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:01.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:01.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:01.475+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:01.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:01.476+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:01.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:01.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:01.311+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:01.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:01 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:01.312+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:02.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:01 smithi055 podman[94542]: 2024-02-20 09:37:01.998991263 +0000 UTC m=+5.773301995 container died cbfa0b8c0da13f3afd6dd2c7261bbb8062b1d5480eefb7138bbdaad69cc6ff49 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-20T09:37:02.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:02 smithi055 podman[94542]: 2024-02-20 09:37:02.386261657 +0000 UTC m=+6.160572390 container cleanup cbfa0b8c0da13f3afd6dd2c7261bbb8062b1d5480eefb7138bbdaad69cc6ff49 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-20T09:37:02.674 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:02 smithi055 bash[94542]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0 2024-02-20T09:37:02.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:02.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:02.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:02.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:02.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:02.434+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:02.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:02.435+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:02.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:02.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:02.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:02.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:02.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:02.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:02.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:02 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:02.346+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:02.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:02 smithi055 podman[94565]: 2024-02-20 09:37:02.73002156 +0000 UTC m=+0.724098757 container remove cbfa0b8c0da13f3afd6dd2c7261bbb8062b1d5480eefb7138bbdaad69cc6ff49 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, org.label-schema.build-date=20240214) 2024-02-20T09:37:03.352 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:03 smithi055 podman[94655]: 2024-02-20 09:37:02.985891041 +0000 UTC m=+0.020311791 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-20T09:37:03.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:03 smithi055 podman[94655]: 2024-02-20 09:37:03.10603229 +0000 UTC m=+0.140453028 container create f66a1aafbdb61f58d30e357f0f2cef6b4b094ce059cbfe2f83625584a0261f1b (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-20T09:37:03.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:03 smithi055 podman[94655]: 2024-02-20 09:37:03.351264417 +0000 UTC m=+0.385685173 container init f66a1aafbdb61f58d30e357f0f2cef6b4b094ce059cbfe2f83625584a0261f1b (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716) 2024-02-20T09:37:03.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:03 smithi055 podman[94655]: 2024-02-20 09:37:03.354816049 +0000 UTC m=+0.389236796 container start f66a1aafbdb61f58d30e357f0f2cef6b4b094ce059cbfe2f83625584a0261f1b (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-20T09:37:03.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:03 smithi055 podman[94655]: 2024-02-20 09:37:03.451463069 +0000 UTC m=+0.485883814 container attach f66a1aafbdb61f58d30e357f0f2cef6b4b094ce059cbfe2f83625584a0261f1b (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-c80e716) 2024-02-20T09:37:03.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:03.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:03.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:03.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:03.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:03.412+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:03.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:03.413+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:03.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:03.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:03.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:03.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:03.687 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:03.355+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:03.688 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:03 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:03.355+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:03.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:03 smithi055 podman[94655]: 2024-02-20 09:37:03.684772975 +0000 UTC m=+0.719193714 container died f66a1aafbdb61f58d30e357f0f2cef6b4b094ce059cbfe2f83625584a0261f1b (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, RELEASE=quincy-c80e716, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-20T09:37:04.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:04 smithi055 podman[94680]: 2024-02-20 09:37:04.322587385 +0000 UTC m=+0.633028995 container remove f66a1aafbdb61f58d30e357f0f2cef6b4b094ce059cbfe2f83625584a0261f1b (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux ) 2024-02-20T09:37:04.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:04 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-20T09:37:04.636 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:04 smithi055 systemd[1]: Stopped Ceph osd.0 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:37:04.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Feb 20 09:37:04 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.0.service: Consumed 47min 49.749s CPU time. 2024-02-20T09:37:04.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:04.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:04.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.419+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:04.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.420+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:04.638 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 systemd[1]: Stopping Ceph osd.1 for 9a1162a4-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:37:04.638 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:04.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:04.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:04.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:04.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:04.347+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:04.639 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:04.347+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.809+0000 7f0db6ba7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:37:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.809+0000 7f0db6ba7700 -1 osd.1 46 *** Got signal Terminated *** 2024-02-20T09:37:04.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:04 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:04.809+0000 7f0db6ba7700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-20T09:37:05.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:05.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:05.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:05.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:05.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:05.345+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:05.620 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:05.346+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:05.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:05.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:05.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:05.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:05.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:05.427+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:05.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:05 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:05.428+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:06.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:06.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:06.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:06.479+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:06.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:06.479+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:06.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:06.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:06.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:06.333+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:06.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:06 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:06.334+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:07.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:07.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:07.443+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:07.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:07.443+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:07.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:07.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:07.336+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:07.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:07 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:07.337+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:08.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:08.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:08.298+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:08.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:08.299+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:08.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:08.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:08.453+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:08.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:08 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:08.454+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:09.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:09.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:09.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:09.482+0000 7f0db21a3700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:36.781403+0000 (oldest deadline 2024-02-20T09:34:56.781403+0000) 2024-02-20T09:37:09.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1[55940]: 2024-02-20T09:37:09.482+0000 7f0db21a3700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:283 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189167b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:37:09.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:09.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:09.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:09.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:09.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:09.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:09 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:09.285+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:10.616 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:10.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:10.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:10.318+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:10.617 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:10 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:10.319+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:10.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:10 smithi055 podman[94759]: 2024-02-20 09:37:10.615297682 +0000 UTC m=+5.920671463 container died eb82fe3e8a01ddca6266f212560f4d2844611275ed3bd1d3a57f39673588f366 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716) 2024-02-20T09:37:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94759]: 2024-02-20 09:37:11.011122431 +0000 UTC m=+6.316496258 container cleanup eb82fe3e8a01ddca6266f212560f4d2844611275ed3bd1d3a57f39673588f366 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-20T09:37:11.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 bash[94759]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1 2024-02-20T09:37:11.625 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94788]: 2024-02-20 09:37:11.287562033 +0000 UTC m=+0.671498399 container remove eb82fe3e8a01ddca6266f212560f4d2844611275ed3bd1d3a57f39673588f366 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux ) 2024-02-20T09:37:11.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:11.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:11.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:11.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:11.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:11.344+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:11.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:11 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:11.344+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:11.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94881]: 2024-02-20 09:37:11.527684161 +0000 UTC m=+0.017830096 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-20T09:37:11.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94881]: 2024-02-20 09:37:11.647011873 +0000 UTC m=+0.137157792 container create 8e91d71502faf58848eca44d662cd85b962e387a0c3c8b57ec7f8525c8bdfb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214) 2024-02-20T09:37:11.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94881]: 2024-02-20 09:37:11.882744662 +0000 UTC m=+0.372890581 container init 8e91d71502faf58848eca44d662cd85b962e387a0c3c8b57ec7f8525c8bdfb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-20T09:37:12.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94881]: 2024-02-20 09:37:11.885792983 +0000 UTC m=+0.375938913 container start 8e91d71502faf58848eca44d662cd85b962e387a0c3c8b57ec7f8525c8bdfb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-20T09:37:12.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:11 smithi055 podman[94881]: 2024-02-20 09:37:11.985227264 +0000 UTC m=+0.475373191 container attach 8e91d71502faf58848eca44d662cd85b962e387a0c3c8b57ec7f8525c8bdfb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux ) 2024-02-20T09:37:12.472 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:12 smithi055 podman[94881]: 2024-02-20 09:37:12.179436962 +0000 UTC m=+0.669582890 container died 8e91d71502faf58848eca44d662cd85b962e387a0c3c8b57ec7f8525c8bdfb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-20T09:37:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:12.294+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:12.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:12.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:12.295+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:12.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:12 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:12.295+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:13.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:12 smithi055 systemd[1]: Stopping Ceph osd.2 for 9a1162a4-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:37:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:12 smithi055 podman[94907]: 2024-02-20 09:37:12.789811873 +0000 UTC m=+0.605179813 container remove 8e91d71502faf58848eca44d662cd85b962e387a0c3c8b57ec7f8525c8bdfb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-02-20T09:37:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:12 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-20T09:37:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:12 smithi055 systemd[1]: Stopped Ceph osd.1 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:37:13.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Feb 20 09:37:12 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.1.service: Consumed 48min 46.264s CPU time. 2024-02-20T09:37:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f150cd6c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:37:13.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f150cd6c700 -1 osd.2 46 *** Got signal Terminated *** 2024-02-20T09:37:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f150cd6c700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-20T09:37:13.723 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:13 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:13.284+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:14.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:14.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:14.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:14.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:14.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:14.253+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:14.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:14 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:14.253+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:15.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:15.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:15.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:15.272+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:15.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:15 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:15.272+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:16.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:16.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:16.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:16.291+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:16.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:16 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:16.291+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:17.721 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:17.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:17.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:17.266+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:17.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:17 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:17.266+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:18.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:18.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:18.225+0000 7f1507b67700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.186:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:34:47.830827+0000 (oldest deadline 2024-02-20T09:35:07.830827+0000) 2024-02-20T09:37:18.722 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:18 smithi055 ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2[63711]: 2024-02-20T09:37:18.226+0000 7f1507b67700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24301.0:1929 3.1e 3:79b2d390:::100000020a1.00000000:head [write 0~2499 in=2499b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-02-20T09:37:19.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:18 smithi055 podman[94985]: 2024-02-20 09:37:18.774624978 +0000 UTC m=+5.632020938 container died 4203cbabc127d986fe7e44a49acc540e4e13c40512c5a9a80f298e1168404ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-20T09:37:19.398 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 podman[94985]: 2024-02-20 09:37:19.145267567 +0000 UTC m=+6.002663527 container cleanup 4203cbabc127d986fe7e44a49acc540e4e13c40512c5a9a80f298e1168404ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS) 2024-02-20T09:37:19.399 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 bash[94985]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2 2024-02-20T09:37:19.718 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 podman[95005]: 2024-02-20 09:37:19.396773809 +0000 UTC m=+0.615758330 container remove 4203cbabc127d986fe7e44a49acc540e4e13c40512c5a9a80f298e1168404ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-20T09:37:19.971 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 podman[95099]: 2024-02-20 09:37:19.619794948 +0000 UTC m=+0.017690538 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-20T09:37:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 podman[95099]: 2024-02-20 09:37:19.731032317 +0000 UTC m=+0.128927886 container create 54f17511085dfd68946df06f24c81e6f37b1a048b85444ee3a3afec6dde0e0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-20T09:37:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 podman[95099]: 2024-02-20 09:37:19.958803369 +0000 UTC m=+0.356698940 container init 54f17511085dfd68946df06f24c81e6f37b1a048b85444ee3a3afec6dde0e0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-02-20T09:37:19.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:19 smithi055 podman[95099]: 2024-02-20 09:37:19.961785149 +0000 UTC m=+0.359680720 container start 54f17511085dfd68946df06f24c81e6f37b1a048b85444ee3a3afec6dde0e0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-c80e716) 2024-02-20T09:37:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:20 smithi055 podman[95099]: 2024-02-20 09:37:20.042101828 +0000 UTC m=+0.439997407 container attach 54f17511085dfd68946df06f24c81e6f37b1a048b85444ee3a3afec6dde0e0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-20T09:37:20.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:20 smithi055 podman[95099]: 2024-02-20 09:37:20.255930439 +0000 UTC m=+0.653826009 container died 54f17511085dfd68946df06f24c81e6f37b1a048b85444ee3a3afec6dde0e0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-20T09:37:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:20 smithi055 podman[95126]: 2024-02-20 09:37:20.837895501 +0000 UTC m=+0.576645601 container remove 54f17511085dfd68946df06f24c81e6f37b1a048b85444ee3a3afec6dde0e0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-02-20T09:37:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:20 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-20T09:37:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:20 smithi055 systemd[1]: Stopped Ceph osd.2 for 9a1162a4-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:37:21.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Feb 20 09:37:20 smithi055 systemd[1]: ceph-9a1162a4-cfc4-11ee-95bb-87774f69a715@osd.2.service: Consumed 47min 57.755s CPU time. 2024-02-20T09:37:27.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:37:27.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:30.208 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:37:39.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:37:39.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:42.272 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:37:54.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:37:54.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:57.345 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:00.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:00.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:00.412 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:00.412 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:38:00.413 INFO:tasks.cephadm:Teardown complete 2024-02-20T09:38:00.413 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:38:00.437 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-20T09:38:00.450 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-20T09:38:00.465 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-20T09:38:00.466 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T09:38:00.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:00.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:03.552 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:12.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:12.554 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:15.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:27.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:27.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:30.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:33.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:33.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:33.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:33.756 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:38:33.803 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:38:33.825 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-20T09:38:33.836 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-20T09:38:33.837 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T09:38:33.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:33.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-02-20T09:38:33.863 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T09:38:33.864 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-02-20T09:38:33.864 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +503us[ +454us] +/- 133ms 2024-02-20T09:38:33.864 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 8 377 25 -866us[ -916us] +/- 74ms 2024-02-20T09:38:33.867 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 229 +850us[ +800us] +/- 90ms 2024-02-20T09:38:33.867 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T09:38:36.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:38:45.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:38:45.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:48.992 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:00.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:00.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:04.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:07.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:07.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:07.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:07.132 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:39:07.133 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:39:07.149 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-20T09:39:07.161 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-20T09:39:07.161 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-20T09:39:07.201 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T09:39:07.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:07.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:10.368 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:19.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:19.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:22.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:34.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:34.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:37.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:40.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:40.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:40.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:40.572 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:39:40.573 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:39:40.581 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-20T09:39:40.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-20T09:39:40.659 INFO:teuthology.task.internal:Duration was 7019.892834 seconds 2024-02-20T09:39:40.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-20T09:39:40.718 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-20T09:39:40.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T09:39:40.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:40.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:40.772 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T09:39:43.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:39:52.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:39:52.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:55.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:07.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:07.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:10.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:13.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:13.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:14.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:14.012 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:40:14.013 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:40:14.029 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-20T09:40:14.043 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-20T09:40:14.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T09:40:14.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:14.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:17.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:26.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:26.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:29.216 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:41.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:41.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:44.288 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:47.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:47.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:47.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:47.356 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:40:47.357 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:40:47.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-20T09:40:47.390 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T09:40:47.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:47.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:40:47.425 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-02-20T09:40:50.464 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:40:59.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:40:59.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:02.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:14.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:14.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:17.596 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:20.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:20.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:20.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:20.668 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:41:20.669 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:41:20.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-20T09:41:20.704 INFO:teuthology.task.internal:Transferring archived files... 2024-02-20T09:41:20.705 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543/remote/smithi055 2024-02-20T09:41:20.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T09:41:20.752 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543/remote/smithi186 2024-02-20T09:41:20.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:20.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:23.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:32.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:32.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:35.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:47.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:47.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:50.944 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:53.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:53.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:54.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:41:54.012 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:41:54.013 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:41:54.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-20T09:41:54.041 INFO:teuthology.task.internal:Not uploading archives. 2024-02-20T09:41:54.042 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-20T09:41:54.054 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-20T09:41:54.054 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T09:41:54.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:41:54.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:41:54.076 INFO:teuthology.orchestra.run.smithi055.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 20 09:38 /home/ubuntu/cephtest 2024-02-20T09:41:54.076 INFO:teuthology.orchestra.run.smithi055.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 20 09:40 /home/ubuntu/cephtest/archive 2024-02-20T09:41:54.076 INFO:teuthology.orchestra.run.smithi055.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 07:42 /home/ubuntu/cephtest/archive/syslog 2024-02-20T09:41:54.077 INFO:teuthology.orchestra.run.smithi055.stdout: 265296 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 20 07:42 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T09:41:54.077 INFO:teuthology.orchestra.run.smithi055.stdout: 265295 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 20 07:42 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T09:41:54.077 INFO:teuthology.orchestra.run.smithi055.stdout: 402360 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 09:39 /home/ubuntu/cephtest/archive/audit 2024-02-20T09:41:54.077 INFO:teuthology.orchestra.run.smithi055.stdout: 426294 180 -rw------- 1 ubuntu ubuntu 180961 Feb 20 09:39 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-20T09:41:54.077 INFO:teuthology.orchestra.run.smithi055.stdout: 265380 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 20 07:49 /home/ubuntu/cephtest/cephadm 2024-02-20T09:41:54.078 INFO:teuthology.orchestra.run.smithi055.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-20T09:41:57.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:42:06.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:42:06.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:42:09.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:42:21.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:42:21.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:42:24.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:42:27.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-02-20T09:42:27.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:42:27.328 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-02-20T09:42:27.328 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:42:27.329 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi186 2024-02-20T09:42:27.345 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-20T09:42:27.358 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-20T09:42:27.371 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-20T09:42:27.371 INFO:teuthology.nuke:Checking targets against current locks 2024-02-20T09:42:27.398 DEBUG:teuthology.nuke:shortname: smithi055 2024-02-20T09:42:27.399 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T09:42:27.422 DEBUG:teuthology.nuke:shortname: smithi186 2024-02-20T09:42:27.423 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T09:42:27.426 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:33:48.103889', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhw/JKRnkaJADFLDR3Xa7VLMAIBk8pQK5DHw2bEuYRAlZcpIRX+cmrW8BQEotiKhl8n9TUmjY4AFghGAYsFqGY='} 2024-02-20T09:42:27.448 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:33:48.102433', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6cYsuhI8MozMWzH0Sjph1vRRLMXMVXpIMfcT2j0scvrS/+Bid6b3tOdWStzxUFp6jvMOSl44DdhGZSejjjL5s='} 2024-02-20T09:42:27.475 INFO:teuthology.orchestra.console.smithi055:Power off 2024-02-20T09:42:27.475 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T09:42:27.498 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-02-20T09:42:27.498 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:27.517 INFO:teuthology.orchestra.console.smithi186:Power off 2024-02-20T09:42:27.517 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T09:42:27.526 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-02-20T09:42:27.540 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-02-20T09:42:27.540 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:27.562 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-02-20T09:42:31.527 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:31.563 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:31.638 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-02-20T09:42:31.672 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-02-20T09:42:35.643 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:35.673 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:35.754 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-02-20T09:42:35.781 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-02-20T09:42:35.881 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-02-20T09:42:39.756 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:42:39.866 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-02-20T09:42:39.967 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-02-20T09:42:40.072 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 7019.892833948135 failure_reason: 'Command failed on smithi055 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a1162a4-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c4295564a6b64dc499a0947839ce008f status: fail success: false 2024-02-20T09:42:40.073 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T09:42:40.154 INFO:teuthology.run:FAIL